inclusion “ is, in practice, demanding. Below, we look at why it is so dif¬cult to

stay close to the requirements demanded by probability sampling:

• The researcher must locate an appropriate sample frame, one that lists all of

the people in the population under investigation. (This sampling frame must

be up-to-date, have no omissions, and be free from duplication.)

• From this (best possible) sampling frame, the next task is to select, in a

completely random way, the individuals to be included in the sample, and

then only to interview these selected individuals. That is, no substitutes can

be taken. (If the sampling interval falls on the Archbishop of Canterbury, it is

this person who must be interviewed. It is unacceptable to choose another

senior cleric, even if he is the Pope!)

• If an individual cannot be interviewed, this needs to be recorded, and declared

as part of the survey™s non-response. As indicated earlier, the target is to obtain

interviews with 65% of the population under investigation. (Only at this point

can we safely say that the attitudes and behaviour of those taking part in our

study are broadly similar to those who do not take part in the study.)

Achieving the above conditions for true probability sampling are extremely

demanding and costly. So, as we have already explained, most commercial

134 Establishing the interpretation boundary

market research takes the form of non-probability sampling, most notably

quota sampling.

Non-probability (quota) sampling

Quota sampling involves a researcher ¬rst obtaining up-to-date information about

the population under investigation. ˜Quotas™ to re¬‚ect the characteristics of this

target universe are then set. These quotas determine the type of individuals that

interviewers will be asked to interview. The usual practice is to set ˜interlocked™

quotas, such that interviewers will be asked to ¬nd, for example, males who are

aged 34 to 45, and who are in the, for instance, C1/C2 socio-economic group,

and also live in North London.

This interlocking practice is a way of helping ensure that interviewers

contact interviewees in a de facto ˜random™ way, rather than being able to

pick “ in a predetermined way “ particular individuals who they know meet

their quota requirements. Thus, quota sampling is predicated on the idea that

the haphazard way in which interviewers initially contact people to ¬ll their

(interlocked) quotas, will approximate the true random probability procedures

outlined above.

Quota sampling is ˜validated™ by virtue of the fact that methodological research

tells us that, in the majority of cases, it leads the commercial researcher to

the same result as probability sampling. Methodological research conducted

by commercial market researchers “ whereby issues are investigated using both

probability and non-probability (quota sampling) methods “ indicate that the

less rigorous quota methodology will be an appropriate ¬tness-to-purpose

design for many market research studies. (Although, clearly, for certain assign-

ments requiring precise measurement, more rigorous probability methods will

be required.)

Knowing how to interpret quota sample generated statistics

Misunderstanding abounds when it comes to interpreting statistics drawn from

quota sampling methods. What many analysts do is simply take what we know

about ˜classic™ sampling theory, outlined above, and make the na¨ve assumption

±

that this body of knowledge applies to quota sampling as it stands. This is

acceptable, insofar as it provides a start point for interpreting a particular survey

statistic. However, it is prudent for the analyst to interpret the resulting estimate

of the error margin in a less literal, and more realistic way “ one that re¬‚ects the

˜compromises™ inherent in the quota sampling process.

Thus, we have already established that, even for ˜high speci¬cation™ probability

samples, there is a ˜design factor™ of between 1.5 and 2.0. (That is, we should

multiply the simple error margin by this ¬gure, to arrive at a realistic estimate

of the true statistic.) Given this, it is perhaps a useful rule of thumb to take

any survey statistic generated from a quota sample, and to assume that the

135

Non-probability (quota) sampling

error margins generated by the simple random sampling formula, should, as an

absolute minimum, always be ˜doubled™.

Of course, sampling error still only tells us part of the story. As we have

already stressed, the bigger issue to take into account, when assessing the

representativeness of a sample, is not just the concept of sampling error, but that

of sampling bias: the extent to which we have excluded people from our study.

This is discussed below.

Understanding the true extent of ˜sampling bias™

We have already indicated that, strictly speaking, with a probability sample, a

minimum of 65% of respondents should have taken part in the survey if we are

to feel comfortable about the survey ¬ndings re¬‚ecting the views of the entire

universe: those who did and did not reply to the survey. However, this raises the

question of how do you arrive at an estimate of the response rate from a process

that followed not probability, but quota, sampling principles.

This is an area of much confusion, even among market research practitioners.

So let us explore this issue further and pinpoint some of the misunderstandings

that surround market research suppliers™ use of the term ˜survey response rate™.

The concept of the response rate

Strictly speaking, the term ˜response rate™ should only be used for probability-

based samples, given that this is the only sampling method that starts life with

a list of the respondents (or organizations) to be interviewed. In Table 10.1

we provide a description of the way in which the response rate on such a

probability-based sample should be presented.

Table 10.1 “ De¬ning a ˜true™ survey response rate (from a probability-based

sample)

• Number issued for interview: 1000

• Number deemed as ineligible (e.g. no longer at address): 100

• Effective sample issued: 900 = 100%

(No.) 900

%

• Number who were not contactable: 120 13

• Number who refused: 230 26

• Number who were successfully interviewed: 550 61

• Response rate (i.e. achieved interviews as % of effective 61%

issued sample with full pro¬le of non-response)

As explained, we know, from a body of empirical evidence “ methodological

research conducted on this topic “ that with probability samples, response rates

136 Establishing the interpretation boundary

of 65% or more mean that the attitudes and behaviour of those taking part

in a survey will broadly re¬‚ect the attitudes and behaviour of those who did

not. Another way of putting this is to say that, once a response rate of 65%

(or over) has been achieved, fairly random factors “ that are unrelated to the

topic under investigation “ tend to explain cooperation and non-cooperation in

surveys. However, as the response rate drifts down below 65%, there is the

chance that those who have taken part in the survey could be different from

those who have not taken part. Or, put another way, it means that factors relating

to the very issue one is trying to measure could begin to explain non-cooperation

in the survey. For example, if only people who have managed to get a job after

leaving a training scheme took part in a survey on what they thought about

the course, then the survey would, of course, provide a ¬‚attering (self-selecting)

account of the effectiveness of the training programme.

The concept of the ˜strike rate™

The above account of response rates, as they relate to probability samples, is

comparatively straightforward. The problem starts when market research ¬eld-

work companies “ that most of the time will have employed quota sampling

methods “ decide to describe the ˜success™ they have had with their interview-

ing “ their ˜strike rate™ “ by (mistakenly) referring to this as a ˜response rate™, and

compounding this error by missing out important information for the data analyst

in interpreting this strike rate.

To unpack this issue, let us start by listing out what information should ideally

be provided when citing a strike rate, drawn from a quota sample. This is outlined

in Table 10.2.

Table 10.2 “ The strike rate from a non-probability (quota) sample