Setting quotas by percentage and variance

Background

Quotas are normally set by counts (shown under the Quota column below). For example, suppose you are seeking 1,000 total responses among a population ages 21 to 50, and you want a balance of the younger and older halves of this range. You could set a quota maximum at 500 for ages 21 to 35, and 500 for ages 36 to 50. This would work fine, allowing up to 500 in each category, and begin terminating people once those limits are achieved.

However, this could theoretically allow the first 500 respondents to be all in the older group with 0 in the younger quota group. Having this kind of imbalance might be undesirable, even if only temporary. For example, if a story in the news or on social media related to the survey topic came out, you could wind up with that news only affecting the later arrivals. If those later respondents were all in the younger group, there would be no way to see if the news affected each group differently. Likely the survey results would be skewed. In addition, if one had to shut down the survey early for some reason, one would be left with an imbalance that would be difficult to correct or handle with weighting. For this reason, IntelliSurvey also allows setting quotas by percentage and variance instead of setting an absolute number as the target. 

Costs to consider with this approach

It is important to realize that setting quotas by percentage will tend to throw out more respondents than quotas set by counts. Continuing with the above example, suppose that at a certain point the survey has 200 people in the older group and 100 in the younger group. 

To define a quota by percentage and variance, select the desired quota tile in the Quota Manager applet, click the Gear > Use target percentages, then enter the desired percents under the Target % column. Once the desired percents are entered, the Target Variance will automatically populate with 5%. This is the system default, but can be edited as needed.

In this example, users could instead set target percentages of 50% each for the younger and older groups, with a 5% allowed variance. This will help ensure that the respondents will never get too far out of balance.

If the Compact View is too small, you can use the scroll bar at the bottom of the tile, or you can switch to Full View.

Returning to the original example with a quota set by counts, let's assume we now have 100 respondents allocated to the 21-35 age group, and 200 allocated to the 36-50 age group. Quota caps for each group are set to 500.

Now, also suppose that a new respondent comes through in the older group. In the regular case of using counts, with quotas at 500 for each group, the respondent would be allowed to continue through the survey.

However, if quotas are set by percentage with 50% each and 5% variance, we would terminate the respondent. This demonstrates that with quotas by percentage, some respondents will be removed before we reach our total target numbers.

This is a desirable result if the goal is to keep the sample in balance, but it could be a disadvantage if the goal is to fill all quota groups as quickly as possible - or if we need to successfully capture as many respondents as possible in a hard-to-reach demographic.

How the system determines effective quotas

When quotas are set numerically (by counts), the system does a simple calculation for each respondent – is the total number of expected Completes greater than or equal to the quota for the respondent's category for that quota? If so, remove that respondent and count them as Overquota; otherwise, let them continue.

When quotas are set by percentage and variance, the system does the same sort of calculation, but it must first convert the percentage and variance into a numerical quota. It does so at that moment, for each respondent that comes through. So in a way, quotas by percentage operate dynamically, as if the numerical quota is constantly changing, based on the total number of responses at that time.

How does the system then convert the percentage and variance into an absolute quota number? The basic calculation is simple:

• The total allowed in a quota group is normally: (Percentage + Variance) x Current N.
  By "Current N", we mean: Total Completes + expected In Progress completes (using predictive closure) for regular quotas. For quotabase or click balance quotas (CBQs), the Current N is the number of records that match the quotabase or cbq base filter.

However, there are two exceptions, designed for managing the beginning and end of fielding:

• For current N less than 300, the calculation is: (Percentage x Current N) + (Variance x 300).
  We use 300 as a default basis for variance in order to allow more variance at the beginning of a survey. Note: If total cap is set and less than 300, we use the total cap as the default instead.
• If a total cap is set, the quota is also limited by the Percentage x Total Cap.

Returning again to our younger/older quota groups example, suppose that the survey is just starting, so we have no Completes and no In Progress responses. That means that for each group, the effective quota is:

Quota = (Percentage x Current N) + (Variance x 300)

               (50% x 0) + (5% x 300) = 15.

Therefore, respondents in either category will be allowed through.

Later, suppose we have 225 in the younger group, and 275 in the older group. This means N=500 (ignoring In Progress completes for instructional purposes). Let us further suppose that a new respondent in the older group comes through. The question then becomes, would they be allowed in?

At this point, the effective quota for the older group is:

Quota = (Percentage + Variance) x Current N

            = (50% + 5%) x 500 = 275.

In this case, we have reached our quota, and the new, older group respondent will be marked as Overquota.

Now suppose we get one more younger person to complete.

In that case, if another respondent in the older group were to enter the survey, the quota for the older group would be:

Quota = (Percentage + Variance) x Current N

            = (50% + 5%) x 501 = 275.55.

Since 275 < 275.55, we would let the next person in the older group through.

As a final example, suppose we are near the end of fielding, and we have 480 younger respondents and 500 in the older group. Now a new older respondent arrives. Will they be allowed to continue the survey?

Quota = (50% + 5%) x 980 = 539 

In this case, they will be termed because we set a quota limit to 500 and 500 < 539.