Ssa_11-27-07

Supporting Statement Part A

Request for Generic Clearance:

QUESTIONNAIRE COGNITIVE

INTERVIEWING AND PRETESTING (ARP/DCCPS/NCI)

Contact Information:

Gordon Willis, PhD.

Cognitive Psychologist

Applied Research Program

Division of Cancer Control and Population Sciences

National Cancer Institute

6130 Executive Blvd, MSC 7344

EPN 4005

Bethesda, MD 20892

301-594-6652

301-435-3710 (fax)

[email protected]

October 15, 2007

Table of Contents

Table of Contents ii

List of Attachments iv

A. JUSTIFICATION 1

A.1. Circumstances Making the Collection of Information Necessary 1

A.2. Purpose and Use of Information 8

A.2.1. Purpose and Use of: Development and Testing of Specific

Survey Questionnaires 8

A.2.2. Purpose and Use of: Research on the Cognitive Aspects of

Survey Methodology 15

A.2.3. Purpose and Use of: Research on Human-Computer

Interfaces/ Usability 17

A.2.4. Purpose and Use of: Pilot Household Interviewing 18

A.3. Use of Improved Information Technology and Burden Reduction 20

A.4. Efforts to Identify Duplication and Use of Similar Information 20

A.5. Involvement of Small Businesses and Other Small Entities 21

A.6. Consequences of Collecting the Information Less Frequently 21

A.7. Special Circumstances Relating to 5 CFR 1320.5 22

A.8. Comments in Response to the Federal Register Notice and Efforts to

Consult Outside Agencies 22

A.9. Explanation of Any Payment of Gift to Respondents 24

A.10. Assurances of Confidentiality Provided to Respondents 26

A.10.1. Informed consent for Intensive Interviews 26

A.10.2. Informed consent for Pilot Household Interviews 28

A.10.3. Confidentiality of responses and safeguarding of materials 28

A.10.4. Interviewer Responsibilities 29

A.10.5. Contractor Responsibilities 31

A.11. Justification of Sensitive Questions 31

A.12. Estimates of Annualized Burden Hours and Costs 32

A.13. Estimates of Total Annual Cost Burden to Respondents

and Recordkeepers 34

A.14. Annualized Costs to the Federal Government 34

A.15. Explanation for Program Changes or Adjustments 35

A.16. Plans for Tabulation and Publication and Project Time Schedule 35

A.17. Expiration Date Display Exemption 36

A.18. Exceptions to Certification 36

LIST OF ATTACHMENTS FOR SUPPORTING STATEMENTS A

ATTACHMENT A: Description of cognitive testing: Willis, G.B. (2005). Cognitive Interviewing. In S.J. Best & B. Radcliff, Polling America: An Encyclopedia of Public Opinion, pp. 92-98. Greenwood Press: Westport, CT.

ATTACHMENT B-1: Sample questionnaire instrument to be tested: Initial draft of 2010-2011 Tobacco Use Supplement to the Current Population Survey

ATTACHMENT B-2: Sample questionnaire instrument to be tested: Sample items on Cancer Quality of Life

ATTACHMENT C: Sample advanced letter to respondents, based on that used for the 2005 HINTS

ATTACHMENT D: Letter from NIH Privacy Act Officer

ATTACHMENT E-1: Assurance of Confidentiality and Informed Consent templates for cognitive interviews

ATTACHMENT E-2: Assurance of Confidentiality and Informed Consent templates for focus groups

ATTACHMENT F: Form for Consent for Expanded Use of Video and Audio Recordings

ATTACHMENT G: NCI Office of Human Subjects Research Exemption Approval

ATTACHMENT H: Sample advertisement (for newspaper or flyer)

ATTACHMENT I: Sample screening script for newspaper/flyer recruitment

A. JUSTIFICATION

A.1. Circumstances Making the Collection of Information Necessary

Background and History. In 1983-1984, the Committee on National Statistics conducted a seminar on the Cognitive Aspects of Survey Methodology (CASM) under a grant from the National Science Foundation (NSF). The participants in the CASM I seminar were cognitive psychologists from academic institutions and survey researchers from the National Center for Health Statistics (NCHS) and the Bureau of the Census. The seminar examined a number of cognitive-related methodological proposals that might lead to improvements in the questionnaires and interviewing procedures employed in scientific surveys in general, and in the NCHS National Health Interview Survey (NHIS) as a test case.

Following this seminar, the NSF provided funding to NCHS to investigate how relevant knowledge and techniques in cognitive science could be applied to im­prove health surveys. The project, begun in 1984, was called Laboratory-Based Studies of the Cognitive Aspects of Survey Methodology, and used cognitive psychological methods to study the survey interviewing process. In its final report, NCHS concluded that it is feasible and efficient for Feder­al statistical agencies to conduct qualitative research on the cogni­tive aspects of survey questionnaires. Subsequently, NCHS applied the cognitive research techniques being tested under the grant to develop the 1987 NHIS supplement, a comprehensive set of questions on knowledge, attitudes, and practices regarding cancer risk factors. Cognitive research techniques - now commonly referred to as cognitive interviewing - proved effective for identifying conceptual problems with draft questions. The NCHS project staff concluded from this experience that past questionnaire design procedures were often unable to identify questions that failed to measure what was intended, and that cognitive interviews were effective for identifying these kinds of measurement errors. At that point, a Questionnaire Design Research Laboratory (QDRL) was created at NCHS to provide such testing for surveys on a regular basis, as well as to continue more general research on the survey response process, questionnaire design, and pretesting methodology (OMB No. 0920-0222).

Since the inception of the QDRL, several other Federal agencies, including the Census Bureau and Bureau of Labor Statistics (BLS), have established cognitive laboratories or otherwise developed capacity for the conduct of cognitive interviews. These interviewing activities are currently conducted under Generic OMB Clearance at NCHS (OMB No. 0920-0222), at the Census Bureau (OMB No. 0607-0725), and at the Bureau of Labor Statistics (OMB No. 1220-0141) as cognitive interviewing techniques have been performed almost continuously for the evaluation of numerous survey questionnaires. In fact, one of the major conclusions of a second CASM seminar (CASM II, held in 1997) was that cognitive testing of survey questionnaires has become a standard practice in the Federal government, as well as in private and academic survey research organizations. Generally, testing staff are not the original authors of the survey questionnaires and do not make decisions about the overall content and survey objectives. Rather, they are methodological specialists – either Agency staff members or contracted specialists -- who submit questionnaires to intensive evaluations designed to improve these measures. This work has proven to be effective for enhancing the quality of Federal survey data for over twenty years.

Proposed NCI Generic Clearance request. Given this history of use of pretesting techniques under Generic Clearance, ARP/DCCPS/NCI (to be referred to from this point simply as NCI) proposes to establish its own Generic clearance to facilitate testing of NCI-sponsored surveys. As stated in the OMB document “Questions and Answers When Designing Surveys for Information Collections: What is a generic clearance for pretesting activities?” (at http://www.whitehouse.gov/omb/inforeg/pmc_survey_guidance_2006.pdf):

Agencies that regularly do pretesting and development work for multiple surveys have found it beneficial to obtain a generic clearance specifically for these kinds of studies. Once the overall generic clearance is obtained on the pretesting activities and methods that will be used (e.g., cognitive interviews, focus groups, respondent debriefings, etc.) through the normal clearance process, agencies can submit abbreviated collection requests on the specific questions to be tested and obtain expedited OMB review (often within 10 working days) of the specific study, which can greatly facilitate ongoing and iterative rounds of testing. For example, cognitive laboratories at the Bureau of Labor Statistics, the Bureau of the Census, and the National Center for Health Statistics have these clearances.

In some cases NCI has pre-existing relationships with agencies that already conduct pretesting under such a Generic clearance. In particular, testing of the periodic Cancer Supplement to the National Health Interview Survey has been conducted at the NCHS QDRL (OMB No. 0920-0222); and pretesting of the NCI-sponsored Tobacco Use Supplement to the Current Population Survey (TUS-CPS) has been conducted by the Census Bureau (OMB No. 0607-0725). Increasingly, however, NCI has developed one-time and periodic surveys – such as the Health Information National Trends Survey (HINTS: OMB No. 0925-0538, in 2005) -- that do not involve collaboration by other Federal agencies, or that would benefit from NCI-sponsored pretesting. For these surveys, it would be advantageous to the government if development were to follow a pretesting sequence equivalent to that used at NCHS, the Census Bureau, or BLS.

As such, NCI staff have developed the capacity for the conduct or oversight of such pretesting techniques, through hiring of a senior methodologist who previously conducted questionnaire cognitive pretesting at the NCHS QDRL, who is expert in pretesting techniques, and who has authored a recent book on survey pretesting (Willis, G., 2005, Cognitive Interviewing- A Tool for Improving Questionnaire Design, Sage, Thousand Oaks, CA). Further, NCI has developed a capacity for the conduct of cognitive interviews through training of its staff, and by establishing connections to several contract research organizations that have similar capabilities. Based on these developments, NCI is in a position to further this initiative by establishing a Generic Pretesting capability through the current request. As such, the current request has been modeled after that currently in use by NCHS and approved by OMB (No. 0920-0222), but tailored to NCI activities.

Cognitive interviewing methods. Methods to be used will follow closely those that have been commonly applied to date in the testing of Federal survey questionnaires. Cognitive interviewing techniques – also called intensive interviews -- focus on the use of both think-aloud and on verbal probing. Generally, a volunteer participant is asked to think aloud as he/she answers the questions, and the specially-trained interviewer probes the participant for additional information. The interviews are generally semi-structured; the interviewer uses draft survey questions as a guide, but probes as needed to determine the participant's interpretation of the questions and the recall, and decision processes used to arrive at his/her answers. This method uncovers ambiguities in question wording, participant strategies for dealing with vague questions, or questions that ask for information that is not readily available (see Attachment A for more detailed information).

A variant of this approach is retrospective cognitive interviewing (or debriefing), in which the interviewer first administers the entire draft questionnaire, and then reviews the questions and responses with the participant, probing for reactions to the questions. While less information is gained about the recall techniques used by participants, there is also less deviation from the natural flow of an interview. In most cases, interviews are audiotaped or videotaped (assuming the subject provides appropriate consent), so that the interviewer can concentrate on probing the responses and can analyze content of the collected information later.

Occasionally, focus groups, typically of 5-10 individuals, are used to discuss general concepts that survey questions will focus on. Individual interviews are generally preferable to focus groups for evaluating specific questions because respondents usually respond to surveys individually, and the group dynamic associated with a focus group format can have a strong influence on interpretations and responses (Fowler, F.J. Jr., (1995), Improving Survey Questions: Design and Evaluation, Applied Social Research Methods Series Volume 38, Sage, Thousand Oaks, CA). However, focus groups can sometimes assist questionnaire designers in understanding the relevant background circumstances of various groups of people, and this information can be used to craft questions that better match respondent experiences (Krueger, R. A. (1994). Focus groups: A practical guide for applied research, Sage, Thousand Oaks, CA).

Additional issues arise in computer-assisted and Internet based survey instruments, involving the human-interface design, ease of use, comprehension, privacy, quality of on-line help and efficiency of screen organization (Couper, M., 1999. The Application of Cognitive Science to Computer Assisted Interviewing, in Sirken, M., Hermann, D., Schechter, S., Schwarz, N., Tanur, J., and Tourangeau, R. (eds.), Cognition and Survey Research, Wiley, New York, pp. 277–300). For questionnaires that involve Web-administration, we will rely on techniques very similar to those used for cognitive interviews, but that involve a more technologically-intensive environment (e.g., administration via laptop or desktop computer).

Pilot Household Interviewing: Although the cognitive interviewing methods described above are effective for identifying problems that are missed by traditional field pretests, they are limited because they do not administer questions under actual field-based interviewing conditions. Therefore, further pilot tests conducted within selected households are a vital complement to cognitive interviews. Survey methodologists sometimes conduct small-scale pilot household interviewing at various points in questionnaire development – not for purposes of field data collection and computation of survey estimates -- but rather as a vital step in the questionnaire development sequence. Also, as time and resources allow, researches apply behavior coding to record the behaviors of both interviewers and survey participants in such interviews to allow for systematic analysis (see Fowler, F. J., & Cannell, C. F. (1996), Using behavioral coding to identify problems with survey questions. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research, San Francisco, Jossey-Bass, pp. 15-36). These activities have been used successfully to develop the questionnaires used in previous Federal questionnaires, such as the NCHS National Health Interview Survey (NHIS) Supplements, and the NCI-sponsored Tobacco Use Supplement to the Current Population Survey (OMB No. 0925-0368). NCI therefore proposes to make use of similar activities in the development of future cancer-related surveys.

Generally, pilot interviews for face-to-face surveys are conducted in the participant's household, and pilot interviews for telephone surveys are conducted over the telephone. Professional field interviewers (often, contactor staff interviewers who are enlisted for the tested survey) normally conduct the interviews. A subset of these interviews is usually observed by a survey professional (a Federal staff member or member of the contract staff). As the interviewer conducts the pilot household interview, the observer compiles notes regarding respondent misunderstandings or difficulties in answering, or questions that interviewers have difficulty administering, which help to identify potential question revisions. This practice allows testing of types of individuals who do not ordinarily volunteer for cognitive interviews, and who may be more typical of the usual survey participant; it also provides information collected under field conditions, and is collected early enough to be useful for questionnaire design decisions.

Finally, we recognize that field pretests are a third type of pretesting, generally conducted after questionnaire development steps (whether cognitive interviews and/or pilot household interviews) are completed. Such pretests are a vital component of preparations for a major survey collection and serve as “dress rehearsal” that evaluates questionnaire flow, length, logical progression, and so on. Cognitive interviews and pilot household interviews do not replace full-scale field pretests, but complement them with a greater focus on questionnaire design issues. Full-scale field pretests of large surveys may consist of 300 or more households, and are normally approved through the regular OMB clearance request process for a fielded survey (either as part of the fielded survey, or as a separate request), rather than through Generic clearance. NCI anticipates that full field pretests for large surveys such as the NHIS will continue to be approved by OMB separately from those activities described in the current request.

Data collection for this project is authorized under 42 USC 242k (Section 410 and 412 of the Public Health Service Act). NCI is requesting terms of clearance similar to that previously granted to NCHS and to the Census Bureau. NCI will submit individual collections under this generic three-year clearance to OMB. OMB will normally provide acknowledgement of receipt, or feedback on the individual collections, within 10 working days of the submission. Approval for participant remuneration of respondents will be evaluated on a case-by-case basis.

A.2. Purpose and Use of Information

The purpose and use of collecting this information fall into four categories—the first three of which involve cognitive/intensive interviews, and the fourth relies on pilot testing with behavior coding:

2.1 Development and testing of specific survey questionnaires

2.2 Research on the cognitive aspects of survey methodology

2.3 Research on human-computer interfaces/usability

2.4 Pilot household interviewing

A.2.1. Purpose and Use of: Development and Testing of Specific Survey Questionnaires

This data collection primarily uses cognitive interviewing methodology to identify and correct questionnaire flaws, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the participant, or otherwise contribute to the non-sampling errors of the survey. Attachment A contains a short description (from an encyclopedia of survey research methods) that outlines the contributions of the cognitive interviewing methodology to the questionnaire development process, the methods used at various stages of the process, and the strengths and limitations of this methodology. The methods used will vary depending on the stage of development of the various data collection instruments to be studied. When questions have been used successfully in earlier surveys, testing will evaluate whether the questions function appropriately in the new context. In cases where there is evidence that previously developed questions were not reliable or valid, more extensive evaluation will be conducted. The most extensive questionnaire development activities will be applied to untested draft questions and undeveloped lists of data objectives. Although we cannot anticipate all of our pretesting activities over the next several years, especially because plans for future surveys depend on budget and establishment of priorities, our planning currently anticipates the following specific survey projects (two sample questionnaires, consisting of very early drafts, are included in Attachment B-1 and B-2):

a) The National Health Interview Survey (NHIS) Cancer Modules. Periodically, NCI sponsors modules to be included on the NHIS (A Cancer Control Supplement was first administered in 1987 to one sample adult aged 18 years or older in each household that participated in the NHIS, and additional Modules were subsequently administered in 1992, 2000, and 2005). Tentative plans are currently underway to field a Cancer Module on the NHIS again in 2010. Currently, the NCHS Questionnaire Design Research Laboratory conducts pretesting activities for the NHIS survey (under OMB No. 0920-0222). However, for future NCI-originated Modules, including that to be administered in 2010, it would be advantageous for NCI to conduct preliminary qualitative pretesting, before sending the Cancer Module to NCHS for further evaluation and testing. In this way, it will be possible to provide well-functioning questions earlier in the developmental process, and ease the testing burden on NCHS specifically related to NCI’s items, so that that organization can focus on the larger question of how the new questions function in the context of the whole NHIS questionnaire.

As NCI anticipates fielding additional Cancer Control modules with the NHIS in 2010, we plan to develop and pretest these questions during the period covered by this clearance request. In all cases, NCI staff will work very closely with NCHS staff involved in questionnaire development and pretesting (in particular with Drs. Kristen Miller and Paul Beatty in the Office of Research and Methodology, who are responsible for cognitive testing of the NHIS under OMB No. 0920-0222). We will continually strive to avoid duplication of effort, and develop a joint testing plan that is maximally efficient, cost-effective, and that minimizes respondent burden.

b) The Health Information National Trends Survey (HINTS). NCI’s HINTS survey has been conducted in 2003 and 2005, and is administered primarily by telephone, although mixed-mode data collection may be used for future surveys. The HINTS is unique in that it monitors the public’s needs and interests with respect to health information in general, and specifically that relating to cancer prevention and control. The latest HINTS survey (fielded in 2005) was covered by OMB No. 0925-0538. Although a subset of core HINTS content remains constant between survey periods, HINTS is also meant to provide dynamic information concerning issues of current interest in the field of health communication (for example, as novel communication technologies develop, especially those involving the Internet, new items are developed to capture usage of those information channels). We anticipate that NCI will conduct or sponsor testing of new HINTS questions between 2008 and 2010, for anticipated field administration in 2010. A major objective of this testing is likely to be the evaluation of sources of response error produced through use of a multi-mode approach. In particular, we will be evaluating the consistency and likelihood of cross-mode comparability of survey questions as they are modified from telephone to self-administration. It is likely that this type of conversion produces cognitive challenges to respondents (as the mode is converted from one that is primarily auditory to another that is visual in nature). Cognitive testing of the different versions, under the intended mode, can be illuminating and may lead to the production of question versions that work acceptably across mode. Contingent on approval by OMB, cognitive and Pilot Household Testing contributing to the next cycle of the HINTS survey will be conducted under the terms set forth in this request.

c) Tobacco Use Supplement to the Current Population Survey (TUS-CPS). The National Cancer Institute periodically sponsors the administration of a large-scale population-based tobacco survey within the CPS, which is itself conducted by the Census Bureau for BLS. For the TUS, NCI varies the topics of emphasis as new data collection needs arise (for example, in 2003 NCI developed the TUS Special Cessation Supplement to track tobacco quitting behaviors; the 2006-7 TUS-CPS provided population surveillance data on tobacco use; and a further cycle is planned for 2010-2011). Because new or modified questions developed for the CPS require careful pretesting, we will also conduct cognitive interviewing of draft forms of new questions that are developed, both to determine the functioning of these items, and to ensure that they function appropriately in the context of existing items.

d) National Physician Survey on Diet, Physical Activity, and Weight Control. The initial NCI survey of physician practices is due for administration in late 2007, contingent on (separate) OMB approval. To date, cognitive testing has been limited to nine respondents (conducted under contract by Westat). NCI has developed tentative plans to monitor changes in physician counseling practices in the future, through further survey cycles. As for the HINTS survey, NCI is committed to a dynamic approach that adds and deletes questions as appropriate over time, and that focuses heavily on efforts to maximize survey response rates by adjusting mode of administration. In particular, we anticipate that although current practice for physician surveys emphasizes mailed, paper-based administration, the use of the Internet can be expected to increase significantly as this mode becomes more acceptable to physicians. Therefore, future testing efforts will focus on the effects of conversion of questions to web-administration, and to establishing that the web-based system exhibits adequate usability, such that good-quality information that minimizes respondent burden can be obtained. As such, NCI staff plan to conduct or oversee cognitive and usability testing of the Physician Survey during the period covered by this request.

e) Health-Related Quality of Life/Quality of Care Assessment

A major challenge to NCI and to epidemiologists and researchers is the development of self-report items for use by patients, and members of the general public, relating to Health-Related Quality of Life (HR-QOL), and more specifically to NCI, self-assessed quality of cancer care received by medical providers. Development of such items that are reliable, and that do not place significant burden on respondents, has presented a consistent challenge to practitioners and researchers interested in the development of items and scales that represent these concepts (Lipscomb, J., Gotay, C., & Snyder, C., 2005, Outcomes Assessment in Cancer, Cambridge). Cognitive testing, and pretesting in general, are potentially useful in the development of these items, especially in conjunction with quantitative methods that involve psychometrics and statistical analysis. As such, NCI staff have been actively engaged in the development of item banks of questions focused on Quality of Life/Care, to be used across a range of future investigations and surveys. We anticipate conducting cognitive testing, and perhaps Pilot testing activities, that focus specifically on the qualitative aspects of these items, and that in particular assess whether they are clear and interpreted similarly across individuals, across patient groups, and across racial/ethnic/cultural groups. The results of these pretesting activities may not be targeted toward a specific survey, but rather toward the establishment of scales that are appropriate for incorporation into future studies. These efforts should greatly facilitate the development of new surveys, as much of the requisite evaluative work will have already been conducted.

f) Other questionnaire testing and development: In addition to the specific questionnaire testing and development activities listed above, we anticipate that NCI staff will perform testing of other questionnaires that require development over a short time-frame. Because the requests may arrive with little advance notice, we cannot presently specify the nature of these questionnaires. As a precedent, such a general plan was used in the NCHS Generic clearance (No. 0920-0222), and has been efficient in allowing NCHS to conduct quick response testing of several questionnaires, including: (1) a CDC Household Drinking Water Internet and Phone Survey, (2) Examination of NHIS questions on race and ethnicity, (3) a FDA Health and Diet Survey, (4) a NCHS U.S.-Canada Joint Health Survey, and (5) a Blood Donor History Screening Questionnaire redesigned by an interagency task force. According to NCHS staff, no procedural problems were encountered as a result of such testing. In fact, the flexibility associated with cognitive interviewing allowed for survey pretesting in a timely manner that minimized participant burden. The field administration of such surveys (as opposed to pretesting of them) will remain dependent on separate, specific OMB approval of data collection activities.

The interviews for questionnaire development activities (a) through (f) above will usually be conducted using procedures described in Attachment A. Interviews are normally conducted at NCI facilities (e.g. at the NCI Usability Laboratory) or in contractor offices (such as the Westat cognitive laboratory). If we are unable to obtain adequate numbers of individuals from particular population subgroups (e.g., the elderly, or those who have specific health problems), we will attempt to make arrangements with organizations such as centers for the elderly, or service organizations for persons with specific health conditions, in order to interview participants at outside locations. Usually, cognitive interviews will be conducted in the mode intended for the survey (face-to-face, telephone, self-administered, or web-based). For a telephone interview, we will either make arrangements to call the participant at home, or to conduct the interview in our laboratory, but calling the participant from another room for questionnaire administration, followed by face-to-face debriefing.

A.2.2. Purpose and Use of: Research on the Cognitive Aspects of Survey Methodology

The second major purpose of data collection is to conduct several cognitive research projects:

a) Respondent perceptions of confidentiality and survey participation: To encourage participation and obtain acceptable survey response rates, NCI surveys such as the HINTS depend on advance letters that promise confidentiality and explaining uses of the data collected (for example, a letter based on the 2005 HINTS advance letter is included as Attachment C). However, it is not known how well these statements are generally understood, and believed, by survey participants. Therefore, NCI staff proposes to conduct cognitive interviews to examine comprehension of such statements. The results will be used to propose modifications to procedures used to communicate key issues related to informed consent, and to explain the need and purpose for survey data in a way intended to increase survey participation.

b) Cross-cultural research. NCI endeavors to conduct basic studies of how best to measure increasingly important factors associated with the cross-cultural aspects of survey response, such as measurement of respondent acculturation. Such questions are key to understanding language and cultural issues that impact access to care, and health in general. NCI staff intend to conduct cognitive testing of these questions, or of newly developed alternative approaches, of Hispanics who are of varying levels of acculturation to U.S. society, to determine whether the questions (in English, and in Spanish) are both understandable, and obtain the types of information intend. Further, we anticipate the development of acculturation questions appropriate as well to Asian or other respondents, and plan to be prepared to develop and evaluate appropriate measures.

c) General methodological research: NCI staff constantly evaluate and refine cognitive interviewing methods, especially in order to respond to changes such as the wide-spread introduction of CAPI (Computer Assisted Personal Interviewing), as data-collection mechanisms for surveys such as the NHIS. Further, NCI staff regularly conduct applied research on questionnaire design issues, such as the optimal wording for measures of complex concepts related to cancer risk factors and related issues (e.g., physical activity; diet; tobacco use; cancer screening). In 2008-2010 NCI staff plan to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, NCI staff and contactors will work collaboratively with survey researchers from Universities and other federal agencies to define and examine several research areas, including, but not limited to: 1) differences between face-to-face and telephone-based cognitive interviewing, 2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing, and 3) social, cultural and linguistic factors in the question response process.

Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, different versions of a survey question will be developed, and the variants then administered to separate groups of participants in order to study the cognitive processes that account for the differences in responses obtained across different versions. These studies will be conducted either by NCI personnel or by contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.

A.2.3. Purpose and use of: Research on Human-Computer Interfaces/Usability

The third major purpose of this data collection is to conduct research on computer-user interface designs for computer-assisted and Web-based instruments, which is often referred to as usability testing. This research examines how survey questions, instructions, and supplemental information are presented on computer instruments, especially over the Internet, and investigates how the presentation affects the ability of users to effectively utilize these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on. Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task (Couper, M., 1999, The Application of Cognitive Science to Computer Assisted Interviewing, in Sirken, M., Hermann, D., Schechter, S., Schwarz, N., Tanur, J., and Tourangeau, R. (eds.), Cognition and Survey Research, New York, Wiley, 277–300). For example, an NCI instrument currently under development – the ASA-24 dietary assessment -- involves Internet-based self-administration, and puts a heavy demand on system usability.

We anticipate an active program of usability testing of such initiatives, based on qualitative assessment that is very similar to the cognitive interviewing of paper-based questionnaires. Usability testing has many obvious similarities to questionnaire-based cognitive research (described in Section A.2.1), as it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat divergent in the sense that dynamic visual information is of greatly increased importance. In particular, it also focuses more heavily on matters of formatting and presentation of information than does traditional cognitive testing.

A.2.4. Purpose and Use of: Pilot Household Interviewing

The activities described above – cognitive interviews, focus groups, and usability studies – can together be terms as intensive interviewing methods. The fourth major purpose of data collection differs from these, as it instead applies to unobtrusive field-based questionnaire evaluation techniques, with respect to future surveys conducted either within the household or over the telephone. The use of the Pilot Household Interview, subsequent to cognitive testing, was introduced by NCHS researchers in the 1990s (NCHS Cognitive Methods Staff Working Papers #3; #15; Hyattsville, MD), and have been supported under OMB Clearance No. 0920-0222). The tested questionnaires may be pilot-tested either individually or in combination, depending on developmental status of the instruments, the appropriateness of combining them, and their overall length. It is envisioned that for any single pilot test, four or five professional field interviewers will conduct a total of approximately 100 pilot household interviews. There are three components to the proposed form of testing: a) a limited number of interviews on a draft version of the questionnaires that are conducted using household participants, by NCI and other staff trained in observational techniques; b) inclusion in the questionnaires of two different versions of particular questions, to gather information relevant to determining which version functions better in the field environment; and c) to make such determinations, the systematic coding of interviewer-respondent interactions.

Overall, the four major activities outlined above have well-demonstrated practical utility. As a result of pretesting, questionnaires may produce substantially less response error than would occur in the absence of this testing. Thus, users of NCI data, in both Federal agencies and in the general health research community, will be less likely to be misled by erroneous statistical results. This assertion is supported by over twenty years of experience in using these techniques, and has been supported by findings presented at many statistical and research related conferences, and published in scientific journals such as Public Opinion Quarterly and Applied Cognitive Psychology. The practical utility of Pilot Household Interviewing has also been supported in findings reported at an annual meeting of the American Statistical Association and the American Association for Public Opinion Research. Further evaluation of the efficacy of these methods will be ongoing.

For later discussion, the term Intensive Pretesting will be used as a general term to refer to cognitive interviews, focus groups, and usability testing; these are distinguished from observational, field-based Pilot Household Interviews.

A.3. Use of Improved Information Technology and Burden Reduction

Pretesting will be conducted using most recent modes of survey data collection, including CAPI/CASI, touch-tone data entry (TDE), the Internet, or other modes applied to Federal surveys.

A.4. Efforts to Identify Duplication and Use of Similar Information

NCI staff work closely with staffs of other Federal agencies that conduct pretesting activities, including (a) the QDRL at NCHS, (b) The Census Bureau’s Center for Survey Methods Research), and (c) the BLS Collection Procedures Research Laboratory. Further, participation of key NCI staff in the Interagency Response Error Group (IREG), which meets quarterly to discuss questionnaire development and pretesting among Federal statistical and research agencies, ensures that pretesting is conducted in a manner that is coordinated across agencies. NCI staff will avoid the conduct of pretesting activities that are duplicative of those of the other agencies. In most cases this is true simply because the various agencies evaluate different survey questionnaires (generally those they develop and administer). However, where surveys do overlap between Agencies (such as the National Health Interview Survey, where responsibility for Cancer-related Modules is shared between NCI and NCHS), NCI staff will collaborate regularly with the other Agency to produce a pretesting plan that is optimal for purposes of timely and efficient production of results, in a way that minimizes respondent burden. This could involve sharing of pretesting responsibilities, but in a coordinated, non-duplicative manner. In some cases parallel testing of the same questions could be conducted across Agencies, for purposes of comparison of pretest results. In particular, as cognitive techniques have been applied, there has been a paucity of research concerning the reliability of obtained results (Beatty & Willis, The Practice of Cognitive Interviewing, Public Opinion Quarterly, 2007); parallel testing between agencies provides an important methodological bridge to provide an answer to this persistent question.

Overall, NCI questionnaire design researchers also maintain very close contact with other experts in the field of questionnaire development in the academic survey community, in the health sciences field, at the Census Bureau, BLS, NCHS, the General Accounting Office, the National Science Foundation, and the Energy Information Administration. From these contacts, it is clear that no other projects that duplicate the current proposal are now underway.

A.5. Involvement of Small Businesses and Other Small Entities

It is possible that representatives of small businesses will be interviewed as part of testing involving medical offices and other establishments, for example as part of the National Physician Survey on Diet, Physical Activity, and Weight Control. For these interviews, the organization/office will be approached in the same manner as the individuals we normally recruit; we will ask the organization to identify the appropriate staff members with whom to conduct the cognitive interview.

A.6. Consequences of Collecting the Information Less Frequently

The project involves one-time data collection activities only. There are no legal obstacles to reducing the burden.

A.7. Special Circumstances Relating to 5 CFR 1320.5

There are no special circumstances.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies

A 60-Day Federal Register notice for this collection was published on November 26, 2007 (Vol. 72, No. 226, p. 65969). No comments have been received. The text of the notice is contained in a file submitted with package titled 60-Day FRN.

Other agencies: Some of the topics selected for NCI surveys are to be developed in conjunction with other agencies: For example, the Tobacco Use Supplement to the Current Population Survey TUS-CPS) is developed in conjunction with the Office of Smoking and Health within the Centers for Disease Control and Prevention. These agencies will be involved in development of draft questionnaires. Further, NCI staff maintain ongoing connections with staffs of NCHS and of the Census Bureau, concerning the development and pretesting of the NHIS, the TUS-CPS, and other joint efforts. With respect to the development of this clearance request, NCI staff have consulted with Karen Whitaker and Kristen Miller of NCHS, in particular, to develop the request in a manner consistent with that used according to the current NCHS Generic Clearance (OMB No. 0920-0222).

Other individuals: Researchers who have special interest and expertise in the research areas explored will be contacted as necessary.

Consultants outside of NCI:

Kristen Miller, Ph.D.

Office of Research and Methodology

National Center for Health Statistics

Centers for Disease Control and Prevention

Toledo Rd

Hyattsville, MD

(301) 458-4625

[email protected]

Floyd J. Fowler, Jr., Ph.D.

Center for Survey Research, University of Massachusetts

100 Arlington Street

Boston, Massachusetts 02116

(617) 956-1150

[email protected]

James Scanlon

Director, Division of Data Policy

Office of the Assistant Secretary for Planning and Evaluation

Hubert H. Humphrey Building

200 Independence Ave., S.W., Room 440D

Washington, D.C. 20201

(202) 690-7100

[email protected]

Kerry Levin

Westat

1650 Research Blvd.

Rockville, MD 20850

(301) 738-3563

[email protected]

Consultants within NCI:

All of the following may be contacted at:

6130 Executive Blvd, MSC 7344

EPN 4005

Bethesda, MD 20892-7344

(301) 435-4742

Rachel Ballard-Barbash, MD, MPH [email protected]

Nancy Breen, PhD [email protected]

Anne Hartman, MS [email protected]

Sue Krebs-Smith, PhD [email protected]

Richard Moser, PhD [email protected]

Bryce Reeve, PhD [email protected]

Rick Troiano, PhD [email protected]

Louise Wideroff, PhD [email protected]

Consultation with representatives of those from whom data will be collected will take place in the form of interviews with volunteers to determine the feasibility of collecting the needed data, the most promising approach for data collection, and general attitudes about the participants which might influence data collection.

A.9. Explanation of Any Payment or Gift to Respondents

For intensive forms of interviews (that is, cognitive interviews, focus groups, and usability tests), participants generally receive remuneration, for several reasons:

 Eligibility criteria for participants are usually specific. Some of these criteria are determined by the subject matter of the survey (e.g., questions may be only relevant to people with certain health conditions). The more specific the subject matter, the more difficult it is to recruit eligible participants; payments help to attract them.

 Intensive forms of interviews require an unusual level of mental effort, as participants are asked to explain their mental processes as they hear the question, discuss its meaning and point out any ambiguities, and evaluate the acceptability of response options that are provided.

 Participants are usually asked to travel to a cognitive laboratory or other testing location, which involves transportation and parking expenses. Many participants incur additional expenses due to leaving their jobs during business hours, making arrangements for child care, etc.).

For a standard interviewing project, in which one-hour intensive interviews are conducted at NCI or contractor offices and eligibility requirements are of average complexity, participants will be paid $50.00. The payment may be reduced to an amount no lower than $30.00 if the interview is of shorter duration, or does not require the participant to travel to NCI. Higher payments may be requested for particularly difficult recruitments. For example, in a 1995 study, the NCHS QDRL has reported being unable to find auto mechanics and truck drivers willing to be interviewed for less than $75.

It is important to offer remuneration sufficient to attract the full range of needed participant types for intensive interviewing projects. Inadequate participant recruitment limits the effectiveness of the questionnaire evaluation. In addition, we face competition from other laboratories (public and private) in a highly saturated research area. Sometimes our advertisements are adjacent to ads offering participants substantially higher payments for the same commitment. Requests and justification for remuneration will be included in each individual collection submission.

For activities that are meant to resemble the usual household interview – in particular, Pilot Household Interviewing -- participants will not receive remuneration, given that the methods are meant to replicate usual field conditions, for which survey respondents are normally not provided remuneration. Further, Pilot interviews are conducted in respondent households, so no travel is required, and also take far less than the one hour required for more intensive pretesting activities such as the cognitive interview.

A.10. Assurances of Confidentiality Provided to Respondents

This study is covered by: 1) the NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” (see letter from NIH Privacy Act Officer, Attachment D); 2) The Common Rule (45CFR46); 3) Section 308(d) of the Public Health Service Act; and 4) Title V of the E-Government Act of 2002.

A.10.1. Informed consent for Intensive (cognitive, focus group, usability) interviews

Participants in intensive interviews are recruited by expressing their personal willingness to participate. Potential participants read or hear about the study through media advertisements, flyers, and word-of-mouth, and either call an NCI-supported answering machine number or contact a person coordinating the recruitment. Thus, participation is strictly voluntary and participants are not chosen randomly.

Tested participants read and sign either Attachment E-1 or E-2, the Assurance of Confidentiality and Informed Consent Forms (written at an 8^th grade reading level), when they arrive at the assigned location for their interview. The form states that participation is voluntary, that they are free to terminate the interview at any time, and that if they do so, they will still receive remuneration. The consent form describes the purpose of interview recording, specifies that the recordings may be played for other staff working closely on that project, that voice and face identifiers will remain on the recordings, and that they may be recognized by a staff member viewing or listening to the recordings. Participants are given a copy of the consent form, which contains contact information for the responsible data collection team.

In addition to consenting for the interview to be recorded at the beginning of the interview, a participant may be asked by the interviewer, at the end of the interview, to read and consider signing Attachment F, the Form for Consent for Expanded Use of Video and Audio Recordings. The purpose of this form is to allow for the playing of interview recordings at conferences, meetings, or in the classroom to illustrate particular findings from the interview. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. If the participant does sign this consent form, he/she will be given a copy which contains information about how to contact the responsible researcher.

Activities covered under this clearance are generally considered to be Exempt from IRB review at NIH, and an initial Exemption Approval is contained in Attachment G. However, in some cases (e.g., where sensitive items are asked, or if Minors are to be interviewed), it will be appropriate to submit a request to conduct the research to the NCI Special Studies Institutional Review Board (SSIRB). As such, for each new project, as well as submitting a specific request to OMB (for ten-day review), NCI staff will submit a request for Exemption to the NIH Office of Human Subjects Research (OHSR). If OHSR determines that the data collection involves non-exempt activities, and should be reviewed by the SSIRB, then NCI staff will develop appropriate materials, and will not contact human subjects for that project until the SSIRB has approved the research Protocol. If a contractor is involved in human subjects research activities, that contractor’s IRB will also review that testing project.

A.10.2. Informed consent for Pilot Household Interviews

For Pilot interviews, whether of household and telephone participants, standard operating procedures regarding informed consent specific to the survey being tested will be slightly modified to reflect participation in the testing of survey questions, rather than participation in the actual survey to be field-administered. For example, for testing of an NHIS Cancer Module, we would use consent materials identical to those of the field survey, but modified to indicate the Pilot nature of the interview.

A.10.3. Confidentiality of responses and safeguarding of materials

The key NCI staff or contractor project director is responsible for safeguarding schedules, consent documents, audiotapes and videotapes, questionnaires, and cash payments to participants.

Documents: Potential participants respond to advertisements and flyers (see Attachment H) by leaving a message on an answering machine or reaching a live receptionist (see Attachment I for the recruitment script). Names, telephone numbers, and other information are transcribed from the answering machine onto a list of potential participants. The list of potential participants, interview schedules, the participant information sheet, the signed Assurance of Confidentiality and Informed Consent Form, the Consent for Expanded Use of Video and Audio Recordings are filed in a project folder and locked in a filing cabinet in project coordinator’s office.

Safeguarding of recordings: The assigned project coordinator will label each recording by participant identifier number, date, time, and project title. No other identifying information is labeled on the recording. The recordings are reviewed, stored, and locked in a private office within NCI or a contractor’s offices. One year following the recording of the interview, the lead researcher will determine the status of the project and potential further uses of the recording. Project status is reviewed on an annual basis to determine if the recordings should be maintained or erased.

A.10.4. Interviewer Responsibilities

Upon completion of a cognitive interview, the interviewer is responsible for the questionnaire, any notes written on other pieces of paper, and if created, the interview recording. The interviewer is instructed to lock all materials in his/her work area until all analysis is completed. Recordings are labeled by participant identifier number, date, time, and project title. No other identifying information is labeled on the recording. Once analysis is completed, interviewers are responsible for returning questionnaires and recordings to the project coordinator, who stores the materials in a locked location.

Reports, publications, and presentations: No participant names or other identifying information is included in any reports, publications, or presentations of interview results.

Interviewing outside of Federal facilities¹: Sometimes interviewers must travel to establishments or individuals’ homes in order to conduct interviews. It is the interviewer’s responsibility to take necessary steps to ensure privacy, confidentiality, and safeguarding of materials. Generally interviews will be conducted in private rooms with a closed door. If no private room is available, the participant can select a private area and the interviewer will judge whether the area is sufficient for ensuing privacy. If the interviewer determines that the area is not private and/or soundproof enough, and no alternative area can be provided, the interview is postponed. For those surveys conducted in the participant’s home, the interviewer requests in advance that the participant arrange for privacy. However, interview location within the home is the choice of the participant. In all cases, extreme care is taken with any materials that contain personal identifiers such as the Assurance of Confidentiality and Informed Consent Form. These materials are immediately transported to NCI or contractor offices, where standard procedures are followed for protection of personally identifiable information.

Focus groups: In focus group settings, participants are interviewed together and can hear each other’s comments, statements, and questions. Participants are told in their initial telephone screening interview that they will be participating in a discussion group with other volunteers. Before the group discussion begins, participants sign the Assurance of Confidentiality and Informed Consent Form which is tailored to specify that they will be participating in a focus group. The Assurance of Confidentiality also states that participants will be asked to pick a name and put it on a name tag, and that they do not have to use a real name. The interviewer (usually referred to as a Moderator when conducting a focus group) will instruct the group that the information discussed will be held confidential by NCI staff and should be treated confidentially by all participants. Participants are asked to respect the privacy of the other participants and not to reveal to others what was discussed by the group.

A.10.5. Contractor Responsibilities

When contractors are employed to collect data as part of NCI projects, they are contractually bound by NCI confidentiality provisions, and must submit documentation concerning ­their safeguarding practices to NCI prior to data collection. For any data collection activity, the contractor’s Institutional Review Board will review the data collection plan, and will complete the review and approval process before contact with human subjects is made.

A.11. Justification of Sensitive Questions

The questionnaires currently proposed for study generally do not contain questions that are highly sensitive in nature. There are exceptions, however, and item sensitivity cannot always be predicted (note that one purpose of pretesting is to assess level of sensitivity). Therefore, a major purpose of cognitive and other pretesting of such questions is to determine means for fashioning them – and explanations for their administration-- in such a way that sensitivity is minimized, and responses are valid.

A.12. Estimates of Annualized Burden Hours and Costs

A. Hour Burden Estimates

The average annual participant burden is estimated to be 600 hours. Estimates of participant burden for each of the questionnaire development studies, over the course of data collection, are provided in Table A.12-1. Estimates are based mainly on the practice of conducting one-hour interviews with participants. The estimates cover the time that each participant will spend communicating with the individual serving as the initial point of contact, in answering screening questions and survey questions and, in some cases, being debriefed following the interviews concerning their thoughts about the tested items. In rare cases, the burden may be more than one hour (although not more than 2 hours). Because the time per response are expected to vary, we will select the final sample size for each project in such a way that the total burden hours do not exceed the estimate listed above. For focus groups, the usual amount of time required is 90 minutes (1.5 hours) with instructions and ancillary paperwork processes taking an additional 15-25 minutes.

For all intensive interviewing activities (cognitive, focus group, or usability) conducted at NCI or contractor offices, the time required to travel to the location of the interview is not included in the current burden estimates, because distances and modes of transportation are unknown. No retrieval of information by participants is anticipated; although it is possible that validation of data at some point may require participants to check records, probably those kept at home. In that case, the study will be designed so that the one-hour response time includes record retrieval. All estimates are based on conferring with NCI staff who coordinate or lead the relevant questionnaire development activities, and on previous small-scale pretesting activities (involving samples of less than nine) that have been conducted under NCI auspices.

Table A.12-1 Estimates of Burden Hours

Type of Respondents	Projects	Number of Respondents	Frequency of Responses/ Participant	Average hours per response	Response burden
Questionnaire Development Volunteers	1) Survey questionnaire development:	200	1	1.25 (75 minutes)	250.0
General Volunteers	2) Research on the cognitive aspects of survey methodology	100	1	1.25 (75 minutes)	125.0
Computer User Volunteers	3) Research on computer-user interface design	100	1	1.25 (75 minutes)	125.0
Household Interview Volunteers	4) Pilot Household interviews	200	1	0.5 (30 minutes)	100.0
Total		600			600.0

B. Annualized Costs to Respondents

No costs are anticipated. Payments to participants are designed to compensate them for their effort and any out-of-pocket costs.

A.13. Estimates of Total Annual Cost Burden to Respondents and Recordkeepers

None.

A.14. Annualized Costs to the Federal Government

The cost to the government consists mainly of the salaries of Federal and contract staff who will: (1) recruit, schedule, and interview volunteer participants, and (2) assist in the analysis of the results and recommend changes in questionnaire wording. Total annualized project costs are located in Table A.14-1.

Table A.14-1 Annualized Costs to the Federal Government

Annual costs for NCI staff to plan, conduct, and analyze the outcomes of the questionnaire development activities:	Managerial	0.50 FTE	$40,000
	Professional	0.50 FTE	$60,000
	Support	1.00 FTE	$40,000
Payment of pretest participants (400 @ $50)			$20,000
Payment, under contract, for assistance with pretesting activities/research			$100,000
Travel costs (mainly local travel):			$1,000
Materials for conducting household interviews			$1,000
Recruitment materials: (flyers, newspaper advertisements):			$2,000
TOTAL			$264,000

Travel costs: Most data will be collected in NCI or contractor office space. However, it will be more efficient in certain instances to hold interviews with individuals at other locations (homes, health centers, elderly centers), which will involve minor travel costs. Further, household interviews will require limited numbers of in-person interviews in participant households. Household interviews will be done locally, in order to limit travel costs, unless there is a compelling reason to do otherwise (for example, if participants critical to the study can be interviewed only at a distant location).

A.15. Explanation for Program Changes or Adjustments

This project is new – the change in burden is 600 hours for each of three years, for a total of 1800 hours.

A.16. Plans for Tabulation and Publication and Project Time Schedule

This clear­ance request is for questionnaire development activities to be conducted prior to field administration, and for developmental work that will guide future questionnaire design. The majority of intensive interviewing investigations will be analyzed qualitatively. The survey designers and lab staff serve as interviewers, and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. The results of these investigations will be used primarily to develop reliable survey instruments and methods. For the Pilot Household Interviewing activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews, in order to determine where additional problems occur. In particular, Behavior Coding, which involves the systematic tabulation of several categories of interviewer behavior (e.g., erroneous reading of question) and respondent behavior (e.g., request for repeat or clarification; providing an uncodeable response) will be used to assess problems with survey questions. Because NCI is using state-of-the-art questionnaire development techniques, NCI staff and collaborating contract staff will produce methodological papers which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the cognitive interview and observation coding techniques.

The time schedule of activities will in most cases be linked to the development cycle of surveys to be supported by the pretesting activities described (e.g., the NHIS, HINTS, and TUS-CPS). All activities will be conducted over the period CY 2008 – 2010.

A.17. Expiration Date Display Exemption

Not applicable.

A.18. Exceptions to Certification

Not applicable.

1Off-site interviews fall into two categories. First, on rare occasions, participants who are recruited for lab interviews request that the interview be conducted in their home. Second, we occasionally conduct establishment studies where a visit to the business location is pertinent to data collection.

Page xxvii