Attach A

ATTACHMENT A: Description of cognitive testing:

Reproduced from:

Willis, G.B. (2005). Cognitive Interviewing. In S.J. Best & B. Radcliff, Polling America: An Encyclopedia of Public Opinion, pp. 92-98. Greenwood Press: Westport, CT.

Topic: Cognitive Interviewing

Contributor: Gordon Willis, Ph.D.

National Cancer Institute

National Institutes of Health

6130 Executive Blvd., MSC 7344

EPN 4005

Bethesda, MD 20892-7344

(301) 594-6652

(301) 435-3710 (fax)

Cognitive Interviewing

As the term is applied in the survey methods field, the Cognitive Interview, also referred to as the Intensive interview and the Laboratory interview, is a technique used to empirically study the ways in which individuals mentally process and respond to materials presented to them in either written or auditory form.¹ Although a wide range of such materials have been evaluated through cognitive interviewing, including advance letters to survey respondents, consent forms, and statistical maps and graphs, the method has mainly been applied to the evaluation of survey questions, especially for the pretesting of draft items as part of the questionnaire development process. The rudiments of the cognitive interviewing approach have existed throughout the history of survey methods. However, cognitive interviewing was formalized and popularized as a result of a collaboration between cognitive psychologists and survey researchers commonly referred to as CASM (Cognition and Survey Methodology), a branch of survey methods which proposes that answering survey questions requires respondents to engage in an often complex series of information processing steps.

The most frequently cited information processing model underlying the cognitive interview is that developed by Tourangeau (1984), which stresses four cognitive stages: (a) Comprehension of the survey question; (b) Retrieval from memory of information necessary to answer that question; (c) Decision processes, especially relating to the adequacy of the answer or the potential threat it may pose due to sensitive content; and (d) the Response process, in which the respondent produces an answer that satisfies the task requirements (e.g., matching an internally generated response to one of a number of qualitative response categories on the questionnaire). For example, to answer the health survey item: “Would you say that in the past 12 months your health in general has been excellent, very good, good, fair, or poor?” the respondent must comprehend key elements of the question (“health in general”), retrieve appropriate memories relevant to the reference period asked about (12 months), make a judgment that represents an overall self-assessment of health, and then match that assessment to one of the given response categories (“excellent” – “poor”).

The cognitive model of the survey response process further proposes that questions may exhibit a variety of features that interfere with successful cognitive processing. For example, the question may be too long and therefore difficult to understand, or may request information that the respondent cannot recall. As a result of such deficiencies, questions tend to produce response error, or that component of survey error that is due to deficiencies in respondents’ reports (as opposed to errors due to imperfections in sampling or to non-response). For example, response error occurs when a respondent mistakenly reports having two full-time jobs in the past ten years when she has actually held four, or when she misinterprets a one-to-ten scale assessing her favorability toward a political candidate and mistakenly selects a value opposite the one representing her true attitude.

A major objective of cognitive interviewing is to study the operation of survey response processes and to identify potential sources of response error across a wide range of survey questions, whether autobiographical (involving personal history, behavior, and events), attitudinal (involving opinions and attitudes), or knowledge-based (the assessment of respondent knowledge of factual material). The cognitive interview resembles the field interview in the sense that an interviewer administers survey questions to an individual, who is in turn asked to answer those questions. However, the cognitive interview departs from the field interview in several key ways. In particular, the function of cognitive interviewing is not to obtain quantitative data, but rather to obtain information relevant to the evaluation of tested questions. On the basis of the results of cognitive testing, these questions are usually modified in order to resolve the problems observed. Because the major emphasis is not data collection or statistical power, but rather the evaluation and development of questions prior to field administration, the sample sizes for a set (or “round”) of cognitive interviews is generally small (typically between 8 and 12 individuals, who are normally referred to as “subjects”). In further departure from the random selection procedures of the field survey, cognitive interviewing depends on volunteers who are recruited in order to represent as wide as possible a range of the population to be surveyed. Finally, the cognitive interview is usually conducted by researcher or other individuals specially trained in the use of cognitive interviewing techniques.

The most distinctive feature of the cognitive interview that differentiates it from the regular field interview is the nature of the interchange between the interviewer and the subject. Rather than relying on standard forms of survey interaction involving a question asking-and-answering sequence, the cognitive interview instead makes use of a range of intensive techniques intended to elucidate the cognitive processes that underlie the survey response process, and to identify problems that subjects have in answering the survey questions that otherwise would not be evident. The conduct of the cognitive interview varies depending on variables such as administration mode (in particular, self- versus interviewer administration). In general, however, the cognitive interviewer administers the tested questions and requests that the subject answer these, but also relies on two activities that characterize the cognitive interview: (a) think-aloud and (b) verbal probing techniques. Think-aloud derives from procedures advocated by Ericsson and Simon (1980) for use in psychological laboratory experiments, and requires subjects to verbalize their thoughts as they answer the survey questions. The interviewer prompts the subject as necessary by providing feedback such as “Tell me what you are thinking” or “Keep talking.” The resulting verbatim record of the subject’s verbalizations is then analyzed in order to identify instances in which problems in answering the tested question originate in the cognitive processing chain.

For example, the subject’s verbal protocol might consist of the following:

“Let’s see – the question is asking me about my opinion on some type of new health insurance reform bill – uh, I’m trying to think about what I know about this… I guess there’s been something in the paper but I really haven’t paid much attention. But I think there are way too many people who don’t have health insurance. So I guess I’d have to say I agree it’s probably a good idea… So… yeah, I favor it.”

Based on this think-aloud passage, it appears that the subject is uninformed about the relevant issues at the level the tested question implicitly assumes, and instead spontaneously converts the question into a more general form that he is able to answer. If such behavior were observed across multiple cognitive interviews, the investigators might consider a modified approach, such as first asking if the person had heard of the proposed bill, or by phrasing the question to make it easier to provide a “no opinion” response.

The second major cognitive interviewing procedure, which has increasingly come into prominence, is verbal probing. In contrast to think-aloud, which primarily extends the respondent’s task, verbal probing explicitly enhances the interviewer’s investigative role. In addition to asking the tested survey question, the cognitive interviewer administers specific probe questions that further elucidate subjects’ processing of that question, and that assess the adequacy of the answers that subjects provide. Immediately after the subject answers the tested question, the interviewer may ask probes such as “Tell me more about that”; “What does the term ‘government spending‘ make you think of?”; or “Was that difficult or easy to remember?”. Probe questions are sometimes selected to tap specific cognitive processes (e.g., comprehension probes assess understanding of the question and its component terms; recall probes assess information retrieval). However, in departure from a pure cognitive orientation, probes also induce the subject to elaborate in ways that make clear whether the tested question applies to that individual, appears to produce responses that are consistent with a more thorough explanation the subject provides, or otherwise exhibits logical or other deficiencies.

Verbal probing can be proactive in nature, where probes are developed prior to the interview, based on the anticipation of potential problems. Or, probes may be reactive, where they are unplanned, but instead elicited based on subject behaviors, such as giving an indication that a question is difficult to answer (for example, a long pause). Proactive forms of probing allow the cognitive interviewer to search for covert problems that otherwise do not surface as a result of the interaction between interviewer and subject. However, it is possible that overuse of these probes may tend to produce the appearance of problems that do not truly exist. Reactive probes avoid the production of artificial problems by focusing only on problems that are clearly evident, but may miss instances in which an underlying problem produces no overt indications. In practice, cognitive interviewers often use a hybrid approach consisting of both forms of probing, and develop a cognitive testing plan that makes use of pre-developed probes, but also allows for more open-ended, spontaneous, and emergent forms of probing.

Probes are usually administered concurrently, or during the conduct of the interview, immediately after the subject has answered each tested question. However, researchers sometimes make use of retrospective probes, or those administered as a set after the interview is completed, and that make reference to questions asked earlier in the interview. A main advantage of concurrent probes is that they induce a verbal report at the time the subject answers the tested question, when relevant verbalizable information is likely to be present in memory. Retrospective probes risk the loss of such memories because they stretch the time between question answering and probing, but are more representative of the normal asking-and-answering sequence to be used in the field interview, and also allow the subject to reflect over the entire interview. Again, cognitive interviewing approaches are flexible, and researchers often rely both on concurrent and retrospective probing, depending on the nature of the questionnaire. For self-administered questionnaires, a common procedure is to leave the subject to complete the instrument undisturbed and unprobed, and then to conduct a retrospective review of their answers. Particularly for paper-based self-administered instruments, this practice allows the researcher to determine how well the subject can follow skip patterns and other sequencing instructions, and to subsequently revisit specific items.

Under any form of probing, cognitive interviews produce data in the form of written notes taken by the interviewer during the course of the interview, of notes taken by observers, or of analysis of (audio or video) recordings of the interview. Such analyses sometimes depend on a coding scheme that applies a particular category of outcome to subjects’ behaviors or to interviewer comments (e.g., “vague term in question”). More often, however, data derived from cognitive interviews are qualitative in nature, consisting of written compilations that describe the problems observed on a question-by-question basis, and often suggest modifications that are intended to address these problems. On the basis of such findings and recommendations, the questionnaire designer or team may revise the tested survey questions and then submit these to further iterative rounds of cognitive testing. The use of iterative testing allows the investigators to determine if the attempted solutions have worked as intended without introducing further problems.

As a concrete example of the overall cognitive testing process in the context of question development, consider the question “Do you believe that your Congressional representatives are doing a good job, or not?” The researchers first consider the manner in which respondents are likely to cognitively process the question, and determine whether there are any indications that the question is deficient in some way. Through this prior review process, they develop probes intended to investigate the basis for a reported opinion, such as: “What , to you, is a ‘Congressional representative’?; “Who are your Congressional representatives?” The tested questions are then administered within a series of laboratory-based cognitive interviews of adults who are selected from the general population, and who are reimbursed for their participation. A finding that is typically obtained based on such testing is that a key term is vague; for the above example, subjects may not be clear about whether we are asking about Senators, Representatives, or both. Further, the cognitive interviews may reveal that individuals who exhibit a strong opinion are nevertheless unable to name even a single Congressperson who represents them. On the basis of such results, the designers may conclude that subjects tend to respond in a way that reflects opinions concerning congress in general (e.g., it either does or does not do a good job). Based on such conclusions, the investigators might try other approaches, such as starting with more specific knowledge questions concerning congressional representatives, and then following up with the original question only for those respondents who are suitably knowledgeable. Or, the focus of the question could be changed by instead asking a substitute question concerning more general attitudes toward Congress.

The development sequence represented by this example can be applied across a variety of projects, surveys, and question types: (a) to-be-evaluated questions are appraised for potential problems; (b) probe questions are developed; (c) appropriate subjects are recruited and administered the questionnaire, using a mix of probe types, and perhaps also instructed to think-aloud; (d) the findings of the interviews are used to make judgments concerning the functioning of the questions; (e) the questions are revised and (f) then re-tested through further iterative testing rounds.

Cognitive interviewing is conducted in a variety of environments and by a range of types of researchers. It can be accomplished informally by a lone questionnaire designer to evaluate a single questionnaire, or can be implemented formally within a cognitive laboratory devoted to the ongoing testing and evaluation of surveys. Permanent cognitive laboratories have been established within several large organizations that regularly conduct surveys, such as at the U.S. Bureau of Labor Statistics, U.S. Census Bureau, or National Center for Health Statistics, and at several large contract research organizations. Within a production laboratory environment, the cognitive laboratory may consist of several experienced cognitive interviewers along with support staff who are responsible for matters such as recruitment and logistical support. Further, within a laboratory environment, cognitive interviewing is generally conducted as part of a more extensive pretesting process, which may also involve alternative methods such as Expert Review or Focus Groups (generally done prior to cognitive testing), or Behavior Coding (generally conducted subsequently, during a field pretesting phase).

Although cognitive interviewing has become a common and accepted practice, the exact procedures used vary widely. Some procedural variation is due to the different categories of surveys that are conducted. For example, the cognitive testing of questionnaires used in establishment surveys (those for which the sampled unit is the organization as opposed to the individual) differs somewhat from that oriented to questionnaires intended for use in surveys of household populations. In particular, cognitive testing of establishment surveys tend to examine issues of information storage and retrieval, especially given that relevant information may be distributed among sources and is often record-based, as opposed to existing in the minds of human respondents. Surveys of individuals also exhibit a range of characteristics that influence the nature of cognitive testing. Questions that focus on very sensitive information (e.g., drug use, sexual behavior, or personal income) in particular need to focus heavily on respondent decision processes that influence the likelihood of truthful and accurate responses.

A second source of variation in cognitive interviewing procedures is not specific to the category of survey under development, but is due more to the inherent flexibility of the method, which gives rise to a range of preferred approaches both within and between cognitive laboratories. Practitioners vary widely in the manner in which they conduct the interview, with respect to reliance on think-aloud versus verbal probing, whether probed interviews rely mainly on proactive versus reactive forms of probes, and whether the cognitive interviewers are conducted by researchers who will also analyze the results and recommend modifications, or by a separate interviewing team that will only present the testing results to others for interpretation and subsequent action. Concerning the process of data coding, analysis, and reporting, the range of approaches involves variation in the degree to which interview recordings are reviewed, and in the level of data reduction reflected in cognitive interview outcome reports (e.g., whether findings are presented at the level of the individual interview or are summarized across interviews).

At this time it is not clear which of these approaches are most reliable or valid. However, several evaluation studies have suggested that (a) the major strength of cognitive interviewing is its elucidation of problems related to question meaning and communication; and (b) this method also identifies questions that present deficiencies because they do not apply well to particular varieties of individuals, are logically misdirected, or do not provide information that satisfies the measurement objectives of the survey. In particular, practitioners have recently focused increasingly on cultural as well as cognitive aspects of survey questions. One promising new direction is the use of the cognitive interview to assess the cross-cultural equivalence of survey questions, especially when questionnaires are translated into languages other than English. In this case cognitive interviewing procedures are extended to diverse population sub-groups, and sometimes to different language versions of the tested questions, in order to determine whether these questions function similarly across group or language. As the method is further developed, it is likely that cognitive interviewing will continue to be applied across a wide range of new areas in which survey-related and other materials are evaluated.

For further reading:

Beatty, P. (In press). AThe Dynamics of Cognitive Interviewing.@ In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Questionnaire Development Evaluation and Testing Methods. New York: John Wiley and Sons. This book chapter addresses several questions related to optimal cognitive interviewing practices, and focuses in particular on what cognitive interviewers are actually found to do in the cognitive laboratory.

DeMaio TJ and Rothgeb JM (1996). “Cognitive Interviewing Techniques: in the Lab and in the Field.” In: Schwarz N and Sudman S (eds), Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research (pp.175-195). San Francisco, Jossey-Bass. This chapter provides a general overview of cognitive interviewing in several testing environments.

Ericsson, K. A., & Simon, H. A. (1980). Verbal Reports as Data. Psychological Review, 87, 215-251. A now-classic paper outlining, reviewing, and advocating the use of verbal report methods in psychology.

Forsyth, B. H., & Lessler, J. T. (1991). “Cognitive Laboratory Methods: A Taxonomy.” In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement Errors in Surveys (pp. 393-418). New York: Wiley. The authors provide an overview of a range of pretesting techniques used to evaluate survey questions.

Jobe, J. B., & Mingay, D. J. (1991). Cognition and Survey Measurement: History and Overview. Applied Cognitive Psychology, 5, 175-192. A summary of the history of the CASM movement.

Tourangeau, R. (1984). “Cognitive Sciences and Survey Methods.” In T. B. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.), Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines (pp. 73-100). Washington, DC: National Academy Press. This paper proposed the model of the survey response process that is most often cited as underlying cognitive interviewing.

Willis, G. B., Royston, P., & Bercini, D. (1991). The Use of Verbal Report Methods in the Development and Testing of Survey Questionnaires. Applied Cognitive Psychology, 5, 251-267. This article gives specific examples of how probing is used to find problems in survey questions, and extends the realm of cognitive interviewing to problems that are logical and structural, as opposed to purely cognitive.

On the internet:

A detailed training manual in cognitive interviewing techniques is downloadable from:

http://appliedresearch.cancer.gov/areas/cognitive/guides.html. Click on “Cognitive Interviewing: A How-To Guide” to access the PDF file. Or, contact the author, Gordon Willis, at [email protected].

Gordon Willis, Ph.D. is Cognitive Psychologist at the National Cancer Institute, National Institutes of Health, Rockville, Maryland.

1 The term Cognitive Interview is also used to refer to procedures intended to enhance the retrieval of information by crime eyewitnesses. The primary objective of this application is not to evaluate materials, but to facilitate accurate reporting from memory. For more information on this application see Fisher, R. P., & Geiselman, R. E. (1992), Memory-Enhancing Techniques for Investigative Interviewing: The Cognitive Interview. Springfield, IL: Thomas.

14