Generic- Description of Cognitive InterviewingTesting

Request for Generic Clearance Cognitive Testing of Instrumentation and Materials for the PATH Study (NIDA)

Attachment 2
Description of Cognitive Interviewing/Testing
August 7, 2012

ATTACHMENT 2: Description of Cognitive Interviewing/Testing

Reproduced from:
Willis, G.B. (2005). Cognitive Interviewing. In S.J. Best & B. Radcliff, Polling America: An Encyclopedia
of Public Opinion, pp. 92-98. Greenwood Press: Westport, CT.

Cognitive Interviewing

As the term is applied in the survey methods field, the Cognitive Interview, also referred to as the
Intensive interview and the Laboratory interview, is a technique used to empirically study the ways in
which individuals mentally process and respond to materials presented to them in either written or
auditory form. 1 Although a wide range of such materials have been evaluated through cognitive
interviewing, including advance letters to survey respondents, consent forms, and statistical maps
and graphs, the method has mainly been applied to the evaluation of survey questions, especially for
the pretesting of draft items as part of the questionnaire development process. The rudiments of the
cognitive interviewing approach have existed throughout the history of survey methods. However,
cognitive interviewing was formalized and popularized as a result of a collaboration between
cognitive psychologists and survey researchers commonly referred to as CASM (Cognition and
Survey Methodology), a branch of survey methods which proposes that answering survey questions
requires respondents to engage in an often complex series of information processing steps.
The most frequently cited information processing model underlying the cognitive interview is that
developed by Tourangeau (1984), which stresses four cognitive stages: (a) Comprehension of the
survey question; (b) Retrieval from memory of information necessary to answer that question; (c)
Decision processes, especially relating to the adequacy of the answer or the potential threat it may
pose due to sensitive content; and (d) the Response process, in which the respondent produces an
answer that satisfies the task requirements (e.g., matching an internally generated response to one of
a number of qualitative response categories on the questionnaire). For example, to answer the health
survey item: “Would you say that in the past 12 months your health in general has been excellent,
very good, good, fair, or poor?” the respondent must comprehend key elements of the question
(“health in general”), retrieve appropriate memories relevant to the reference period asked about (12
months), make a judgment that represents an overall self-assessment of health, and then match that
assessment to one of the given response categories (“excellent” – “poor”).
The cognitive model of the survey response process further proposes that questions may exhibit a
variety of features that interfere with successful cognitive processing. For example, the question may
be too long and therefore difficult to understand, or may request information that the respondent

1

The term Cognitive Interview is also used to refer to procedures intended to enhance the retrieval of information by crime eyewitnesses. The
primary objective of this application is not to evaluate materials, but to facilitate accurate reporting from memory. For more information on this
application see Fisher, R. P., & Geiselman, R. E. (1992), Memory-Enhancing Techniques for Investigative Interviewing: The Cognitive Interview.
Springfield, IL: Thomas.

1

cannot recall. As a result of such deficiencies, questions tend to produce response error, or that
component of survey error that is due to deficiencies in respondents’ reports (as opposed to errors due
to imperfections in sampling or to non-response). For example, response error occurs when a
respondent mistakenly reports having two full-time jobs in the past ten years when she has actually
held four, or when she misinterprets a one-to-ten scale assessing her favorability toward a political
candidate and mistakenly selects a value opposite the one representing her true attitude.
A major objective of cognitive interviewing is to study the operation of survey response processes
and to identify potential sources of response error across a wide range of survey questions, whether
autobiographical (involving personal history, behavior, and events), attitudinal (involving opinions
and attitudes), or knowledge-based (the assessment of respondent knowledge of factual material).
The cognitive interview resembles the field interview in the sense that an interviewer administers
survey questions to an individual, who is in turn asked to answer those questions. However, the
cognitive interview departs from the field interview in several key ways. In particular, the function of
cognitive interviewing is not to obtain quantitative data, but rather to obtain information relevant to
the evaluation of tested questions. On the basis of the results of cognitive testing, these questions
are usually modified in order to resolve the problems observed. Because the major emphasis is not
data collection or statistical power, but rather the evaluation and development of questions prior to
field administration, the sample sizes for a set (or “round”) of cognitive interviews is generally small
(typically between 8 and 12 individuals, who are normally referred to as “subjects”). In further
departure from the random selection procedures of the field survey, cognitive interviewing depends
on volunteers who are recruited in order to represent as wide as possible a range of the population
to be surveyed. Finally, the cognitive interview is usually conducted by researcher or other
individuals specially trained in the use of cognitive interviewing techniques.
The most distinctive feature of the cognitive interview that differentiates it from the regular field
interview is the nature of the interchange between the interviewer and the subject. Rather than
relying on standard forms of survey interaction involving a question asking-and-answering sequence,
the cognitive interview instead makes use of a range of intensive techniques intended to elucidate
the cognitive processes that underlie the survey response process, and to identify problems that
subjects have in answering the survey questions that otherwise would not be evident. The conduct
of the cognitive interview varies depending on variables such as administration mode (in particular,
self- versus interviewer administration). In general, however, the cognitive interviewer administers
the tested questions and requests that the subject answer these, but also relies on two activities that
characterize the cognitive interview: (a) think-aloud and (b) verbal probing techniques. Think-aloud
derives from procedures advocated by Ericsson and Simon (1980) for use in psychological
2

laboratory experiments, and requires subjects to verbalize their thoughts as they answer the survey
questions. The interviewer prompts the subject as necessary by providing feedback such as “Tell me
what you are thinking” or “Keep talking.” The resulting verbatim record of the subject’s
verbalizations is then analyzed in order to identify instances in which problems in answering the
tested question originate in the cognitive processing chain.
For example, the subject’s verbal protocol might consist of the following:
“Let’s see – the question is asking me about my opinion on some type of
new health insurance reform bill – uh, I’m trying to think about what I
know about this… I guess there’s been something in the paper but I really
haven’t paid much attention. But I think there are way too many people
who don’t have health insurance. So I guess I’d have to say I agree it’s
probably a good idea… So… yeah, I favor it.”
Based on this think-aloud passage, it appears that the subject is uninformed about the relevant issues
at the level the tested question implicitly assumes, and instead spontaneously converts the question
into a more general form that he is able to answer. If such behavior were observed across multiple
cognitive interviews, the investigators might consider a modified approach, such as first asking if the
person had heard of the proposed bill, or by phrasing the question to make it easier to provide a “no
opinion” response.
The second major cognitive interviewing procedure, which has increasingly come into prominence,
is verbal probing. In contrast to think-aloud, which primarily extends the respondent’s task, verbal
probing explicitly enhances the interviewer’s investigative role. In addition to asking the tested
survey question, the cognitive interviewer administers specific probe questions that further elucidate
subjects’ processing of that question, and that assess the adequacy of the answers that subjects
provide. Immediately after the subject answers the tested question, the interviewer may ask probes
such as “Tell me more about that”; “What does the term ‘government spending‘ make you think
of?”; or “Was that difficult or easy to remember?”. Probe questions are sometimes selected to tap
specific cognitive processes (e.g., comprehension probes assess understanding of the question and
its component terms; recall probes assess information retrieval). However, in departure from a pure
cognitive orientation, probes also induce the subject to elaborate in ways that make clear whether
the tested question applies to that individual, appears to produce responses that are consistent with a
more thorough explanation the subject provides, or otherwise exhibits logical or other deficiencies.

3

Verbal probing can be proactive in nature, where probes are developed prior to the interview, based
on the anticipation of potential problems. Or, probes may be reactive, where they are unplanned,
but instead elicited based on subject behaviors, such as giving an indication that a question is
difficult to answer (for example, a long pause). Proactive forms of probing allow the cognitive
interviewer to search for covert problems that otherwise do not surface as a result of the interaction
between interviewer and subject. However, it is possible that overuse of these probes may tend to
produce the appearance of problems that do not truly exist. Reactive probes avoid the production of
artificial problems by focusing only on problems that are clearly evident, but may miss instances in
which an underlying problem produces no overt indications. In practice, cognitive interviewers often
use a hybrid approach consisting of both forms of probing, and develop a cognitive testing plan that
makes use of pre-developed probes, but also allows for more open-ended, spontaneous, and
emergent forms of probing.
Probes are usually administered concurrently, or during the conduct of the interview, immediately
after the subject has answered each tested question. However, researchers sometimes make use of
retrospective probes, or those administered as a set after the interview is completed, and that make
reference to questions asked earlier in the interview. A main advantage of concurrent probes is that
they induce a verbal report at the time the subject answers the tested question, when relevant
verbalizable information is likely to be present in memory. Retrospective probes risk the loss of such
memories because they stretch the time between question answering and probing, but are more
representative of the normal asking-and-answering sequence to be used in the field interview, and
also allow the subject to reflect over the entire interview. Again, cognitive interviewing approaches
are flexible, and researchers often rely both on concurrent and retrospective probing, depending on
the nature of the questionnaire. For self-administered questionnaires, a common procedure is to
leave the subject to complete the instrument undisturbed and unprobed, and then to conduct a
retrospective review of their answers. Particularly for paper-based self-administered instruments, this
practice allows the researcher to determine how well the subject can follow skip patterns and other
sequencing instructions, and to subsequently revisit specific items.
Under any form of probing, cognitive interviews produce data in the form of written notes taken by
the interviewer during the course of the interview, of notes taken by observers, or of analysis of
(audio or video) recordings of the interview. Such analyses sometimes depend on a coding scheme
that applies a particular category of outcome to subjects’ behaviors or to interviewer comments (e.g.,
“vague term in question”). More often, however, data derived from cognitive interviews are
qualitative in nature, consisting of written compilations that describe the problems observed on a
question-by-question basis, and often suggest modifications that are intended to address these
4

problems. On the basis of such findings and recommendations, the questionnaire designer or team
may revise the tested survey questions and then submit these to further iterative rounds of cognitive
testing. The use of iterative testing allows the investigators to determine if the attempted solutions
have worked as intended without introducing further problems.
As a concrete example of the overall cognitive testing process in the context of question
development, consider the question “Do you believe that your Congressional representatives are
doing a good job, or not?” The researchers first consider the manner in which respondents are likely
to cognitively process the question, and determine whether there are any indications that the
question is deficient in some way. Through this prior review process, they develop probes intended
to investigate the basis for a reported opinion, such as: “What , to you, is a ‘Congressional
representative’?; “Who are your Congressional representatives?” The tested questions are then
administered within a series of laboratory-based cognitive interviews of adults who are selected from
the general population, and who are reimbursed for their participation. A finding that is typically
obtained based on such testing is that a key term is vague; for the above example, subjects may not
be clear about whether we are asking about Senators, Representatives, or both. Further, the
cognitive interviews may reveal that individuals who exhibit a strong opinion are nevertheless unable
to name even a single Congressperson who represents them. On the basis of such results, the
designers may conclude that subjects tend to respond in a way that reflects opinions concerning
congress in general (e.g., it either does or does not do a good job). Based on such conclusions, the
investigators might try other approaches, such as starting with more specific knowledge questions
concerning congressional representatives, and then following up with the original question only for
those respondents who are suitably knowledgeable. Or, the focus of the question could be changed
by instead asking a substitute question concerning more general attitudes toward Congress.
The development sequence represented by this example can be applied across a variety of projects,
surveys, and question types: (a) to-be-evaluated questions are appraised for potential problems;
(b) probe questions are developed; (c) appropriate subjects are recruited and administered the
questionnaire, using a mix of probe types, and perhaps also instructed to think-aloud; (d) the
findings of the interviews are used to make judgments concerning the functioning of the questions;
(e) the questions are revised and (f) then re-tested through further iterative testing rounds.
Cognitive interviewing is conducted in a variety of environments and by a range of types of
researchers. It can be accomplished informally by a lone questionnaire designer to evaluate a single
questionnaire, or can be implemented formally within a cognitive laboratory devoted to the ongoing
testing and evaluation of surveys. Permanent cognitive laboratories have been established within
5

several large organizations that regularly conduct surveys, such as at the U.S. Bureau of Labor
Statistics, U.S. Census Bureau, or National Center for Health Statistics, and at several large contract
research organizations. Within a production laboratory environment, the cognitive laboratory may
consist of several experienced cognitive interviewers along with support staff who are responsible
for matters such as recruitment and logistical support. Further, within a laboratory environment,
cognitive interviewing is generally conducted as part of a more extensive pretesting process, which
may also involve alternative methods such as Expert Review or Focus Groups (generally done prior
to cognitive testing), or Behavior Coding (generally conducted subsequently, during a field pretesting
phase).
Although cognitive interviewing has become a common and accepted practice, the exact procedures
used vary widely. Some procedural variation is due to the different categories of surveys that are
conducted. For example, the cognitive testing of questionnaires used in establishment surveys (those
for which the sampled unit is the organization as opposed to the individual) differs somewhat from
that oriented to questionnaires intended for use in surveys of household populations. In particular,
cognitive testing of establishment surveys tend to examine issues of information storage and
retrieval, especially given that relevant information may be distributed among sources and is often
record-based, as opposed to existing in the minds of human respondents. Surveys of individuals also
exhibit a range of characteristics that influence the nature of cognitive testing. Questions that focus
on very sensitive information (e.g., drug use, sexual behavior, or personal income) in particular need
to focus heavily on respondent decision processes that influence the likelihood of truthful and
accurate responses.
A second source of variation in cognitive interviewing procedures is not specific to the category of
survey under development, but is due more to the inherent flexibility of the method, which gives rise
to a range of preferred approaches both within and between cognitive laboratories. Practitioners
vary widely in the manner in which they conduct the interview, with respect to reliance on thinkaloud versus verbal probing, whether probed interviews rely mainly on proactive versus reactive
forms of probes, and whether the cognitive interviewers are conducted by researchers who will also
analyze the results and recommend modifications, or by a separate interviewing team that will only
present the testing results to others for interpretation and subsequent action. Concerning the
process of data coding, analysis, and reporting, the range of approaches involves variation in the
degree to which interview recordings are reviewed, and in the level of data reduction reflected in
cognitive interview outcome reports (e.g., whether findings are presented at the level of the
individual interview or are summarized across interviews).

6

At this time it is not clear which of these approaches are most reliable or valid. However, several
evaluation studies have suggested that (a) the major strength of cognitive interviewing is its
elucidation of problems related to question meaning and communication; and (b) this method also
identifies questions that present deficiencies because they do not apply well to particular varieties of
individuals, are logically misdirected, or do not provide information that satisfies the measurement
objectives of the survey. In particular, practitioners have recently focused increasingly on cultural as
well as cognitive aspects of survey questions. One promising new direction is the use of the
cognitive interview to assess the cross-cultural equivalence of survey questions, especially when
questionnaires are translated into languages other than English. In this case cognitive interviewing
procedures are extended to diverse population sub-groups, and sometimes to different language
versions of the tested questions, in order to determine whether these questions function similarly
across group or language. As the method is further developed, it is likely that cognitive interviewing
will continue to be applied across a wide range of new areas in which survey-related and other
materials are evaluated.
For further reading:
Beatty, P. (In press). The Dynamics of Cognitive Interviewing. In S. Presser, J. Rothgeb, M. Couper,
J. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Questionnaire Development Evaluation and
Testing Methods. New York: John Wiley and Sons. This book chapter addresses several
questions related to optimal cognitive interviewing practices, and focuses in particular on what
cognitive interviewers are actually found to do in the cognitive laboratory.
DeMaio, T.J., and Rothgeb, J.M. (1996). Cognitive Interviewing Techniques: in the Lab and in the
Field.” In: Schwarz N and Sudman S (eds), Answering Questions: Methodology for Determining
Cognitive and Communicative Processes in Survey Research (pp.175-195). San Francisco, Jossey-Bass.
This chapter provides a general overview of cognitive interviewing in several testing
environments.
Ericsson, K.A., & Simon, H.A. (1980). Verbal Reports as Data. Psychological Review, 87, 215-251. A
now-classic paper outlining, reviewing, and advocating the use of verbal report methods in
psychology.
Forsyth, B.H., & Lessler, J.T. (1991). Cognitive Laboratory Methods: A Taxonomy. In P. P. Biemer,
R.M. Groves, L.E. Lyberg, N.A. Mathiowetz, & S. Sudman (Eds.), Measurement Errors in Surveys
(pp. 393-418). New York: Wiley. The authors provide an overview of a range of pretesting
techniques used to evaluate survey questions.
Jobe, J.B., & Mingay, D.J. (1991). Cognition and Survey Measurement: History and Overview.
Applied Cognitive Psychology, 5, 175-192. A summary of the history of the CASM movement.

7

Tourangeau, R. (1984). Cognitive Sciences and Survey Methods. In T.B. Jabine, M.L. Straf, J.M.
Tanur, & R. Tourangeau (Eds.), Cognitive Aspects of Survey Methodology: Building a Bridge Between
Disciplines (pp. 73-100). Washington, DC: National Academy Press. This paper proposed the
model of the survey response process that is most often cited as underlying cognitive
interviewing.
Willis, G.B., Royston, P., & Bercini, D. (1991). The Use of Verbal Report Methods in the
Development and Testing of Survey Questionnaires. Applied Cognitive Psychology, 5, 251-267.
This article gives specific examples of how probing is used to find problems in survey
questions, and extends the realm of cognitive interviewing to problems that are logical and
structural, as opposed to purely cognitive.
On the internet:
A detailed training manual in cognitive interviewing techniques is downloadable from:
http://appliedresearch.cancer.gov/areas/cognitive/guides.html. Click on “Cognitive
Interviewing: A How-To Guide” to access the PDF file. Or, contact the author, Gordon
Willis, at [email protected].

Gordon Willis, Ph.D., is Cognitive Psychologist at the National Cancer Institute, National Institutes
of Health, Rockville, Maryland.

8