Supporting Statement A for Request for Clearance:
QUESTIONNAIRE DESIGN RESEARCH LABORATORY
2012-2015
OMB No. 0920-0222
Contact Information:
Karen R. Whitaker, B.S.
Laboratory Manager, Questionnaire Design Research Laboratory
Office of Research and Methodology
National Center for Health Statistics/CDC
3311 Toledo Road, Room 3215
Hyattsville, MD 20782
301-458-4569
301-458-4031 (fax)
March 22, 2012
SUMMARY
A three-year OMB clearance revision is requested for “NCHS Questionnaire Design Research Laboratory (OMB No. 0920-0222).” This generic clearance request encompasses general questionnaire development and pre-testing activities to be carried out in 2012-2015 in the Office of Research and Methodology, National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC). The activities are to be conducted by the staff of the Questionnaire Design Research Laboratory (QDRL) and involve the development of health-related survey questionnaires, using a methodology which has been employed effectively since 1985. In 2012, the QDRL will be conducting a large-scale test of questions and procedures in conjunction with the National Health Interview Survey (0920-0214). Details are shown in section A.1. Clearance is requested for a period of 3 years.
Five types of activities are carried out:
1) Survey questionnaire development and testing based on cognitive interviewing methodology. Draft questionnaires to be used in CDC, other federal agencies, or other academic or professional institutions are developed and tested through rounds of cognitive interviews in laboratory or field settings with specifically-recruited volunteer respondents. Most commonly these are one-on-one cognitive interviews, although group interviews (focus groups) are also used occasionally. Results of cognitive interviews are used to make questionnaire design decisions that minimize survey response error.
2) Research on the cognitive and interpretive aspects of survey methodology. Such research could take the form of experiments embedded within fielded surveys (generally referred to as “split-ballot” experiments), experiments conducted in the laboratory, or exploratory studies employing one-on-one cognitive interviews or focus groups. In all cases, the purpose of the research is not necessarily to test particular questions, but to enhance our understanding of the question response process, to develop better standards for questionnaire design, or to improve data collection procedures.
3) Research on computer-user interface design for computer-assisted instruments (including Web-based surveys), also known as usability testing. Laboratory or field respondents help us assess ease of use (e.g. Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self-Interviewing (CASI) instruments), Audio Computer-Assisted Self-Interview (ACASI), comprehension, and quality of on-line help.
4) Field Tests/Pilot interviews: A limited number of pilot interviews (either personal or telephone) are conducted with household respondents, using professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested). Sources of response error are identified through observation by methodologists and techniques such as the coding of the interviewer-respondent interaction.
5) Studies of the optimal design and presentation of statistical graphical and textual material. This could include evaluations of the effectiveness of data presentation on the Web (including Web-based surveys), informational brochures, poster presentations, and statistical publications.
OMB Approval of Individual Collections
CDC will continue to submit individual collections under this generic clearance to OMB. OMB will provide feedback on the individual collections within ten working days, whenever possible, as is currently the case. Standard remuneration of respondents is $40.00. Higher remunerations may be requested with justification for difficult recruitments, focus groups, etc.
SUPPORTING STATEMENT FOR OMB CLEARANCE
NCHS Questionnaire Development and Pre-testing Research
A. JUSTIFICATION
1. Circumstances Making the Collection of Information Necessary
In 1983/1984, the Committee on National Statistics conducted a two-part seminar on the Cognitive Aspects of Survey Methodology (CASM) under a grant from the National Science Foundation (NSF). The respondents in the CASM seminar (CASM I) were survey researchers and cognitive psychologists from academic institutions and survey researchers from the National Center for Health Statistics (NCHS) and the Bureau of the Census. The seminar examined a number of cognitive-related methodological studies that might lead to improvements in the questionnaires and interviewing procedures employed in scientific surveys in general, and in the National Health Interview Survey (NHIS) as a test case.
Following this seminar, the NSF provided funding to NCHS to investigate how relevant knowledge and techniques in cognitive science could be applied to improve health surveys. The project, begun in 1984, was called Laboratory-Based Studies of the Cognitive Aspects of Survey Methodology (CASM), and used cognitive psychological methods to study the survey interviewing process. In its final report, NCHS concluded that it is feasible and efficient for a Federal statistical agency to conduct laboratory research on the cognitive aspects of survey questionnaires. Subsequently, NCHS applied the cognitive research techniques being tested under the grant to develop the 1987 NHIS supplement, a comprehensive set of questions on knowledge, attitudes, and practices regarding cancer risk factors. Cognitive research techniques (also known as cognitive interviewing) proved invaluable for identifying conceptual problems with draft questions. The NCHS project staff concluded from this experience that past questionnaire design procedures were often unable to identify questions that were failing to measure what was intended, but that interviews in the laboratory were effective for identifying these kinds of measurement errors. The Questionnaire Design Research Laboratory (QDRL) was created at NCHS to provide such testing for NCHS surveys on a regular basis, as well as to continue more general research on the survey response process, questionnaire design, and pretesting methodology.
Furthermore, in October 2009, NCHS held a Question Evaluation Methods Workshop to examine various question evaluation methods as well as to discuss the impact of question design and evaluation on survey data quality. Broad consensus determined that measurement error in the Federal statistical enterprise requires renewed consideration. Federal statistical agencies have a fundamental obligation to produce valid and reliable data, and more attention needs to be placed on question evaluation and documentation. Furthermore, it was established that the validation of measures is a particular complex, methodological problem that requires a mixed-method approach. While quantitative methods are essential for understanding the magnitude and prevalence of error, they remain dependent on the interpretive power of cognitive interviewing. Unlike any other question evaluation method, cognitive interviewing can portray the interpretive processes that ultimately produce survey data. As it is practiced as a qualitative methodology, cognitive interviewing reveals these processes as well as the type of information that is transported through statistics. Consequently, cognitive interviewing is an integral method for ensuring the validity of statistical data, and the documented findings from these studies represent tangible evidence of how the question performs.
In 2012, the QDRL has been requested to undertake a project that is substantially larger than its ongoing activities. The ACASI content included in the 5,000-case test is consistent with the content studied in the 575- and 50-case tests. Details on the 50 and 575 case test can be found in 2.1.a. The module includes questions on sexual identity, alcohol consumption, HIV testing, mental health, height and weight, sleep, financial worries, and others (see Attachment P).
The 5,000-case test will include one or more built-in experiments to assess the impact of ACASI, and components of ACASI, on prevalence estimates and data quality. First and foremost, test cases will be randomly assigned to receive the above described questions in either CAPI or ACASI. In particular, prevalence estimates for the sexual identity questions will be compared by mode of administration. Since a documented advantage of ACASI is the enhanced level of privacy it affords, we anticipate higher prevalence estimates of sexual minorities (Lesbian, Gay, Bisexual and Transgender persons) from this mode of administration. Estimates for sensitive items on mental health, alcohol consumption, HIV testing, height and weight, financial worries, and others will also be compared.
In addition, direct and indirect indicators of data quality, such as item nonresponse and response times, can also be compared between the CAPI and ACASI administrations. And since the field test will use a nationally-representative sample, content pulled from the core NHIS instrument and moved to the ACASI module can be compared between the field test NHIS and the on-going, production NHIS. Along with mode effects, this will allow us to explore possible context effects on prevalence estimates, item nonresponse rates, and other quality indicators.
It is the mission of the QDRL to evaluate questions for optimal design, but also to provide documentation supporting the validity of NCHS survey data. Such documentation also serves NCHS data users, allowing them to be critical users in their approach and application of the data. Consequently, all completed QDRL testing reports are located and made accessible on Q-Bank, an interagency, online searchable database that houses question evaluation studies.
Data collection for this project is authorized under 42 USC 242k (Section 306 of the Public Health Service Act). A copy of the legislation is provided in Attachment A. CDC is requesting terms of clearance identical to previous submissions. CDC will submit individual collections under this generic three year clearance to OMB. OMB will continue to provide feedback on the individual collections within 10 working days of the submission. Standard remuneration of respondents is $40.00. Higher remunerations may be requested with justification for difficult recruitments, focus groups, etc.
Privacy Impact Assessment
A Privacy Impact Assessment was submitted on November 29, 2011 as part of the QDRL’s 2011 C&A recertification.
Overview of the Data Collection System
QDRL Staff are methodological specialists who examine questionnaires from NCHS, CDC, other federal agencies, or other academic or professional institutions. Specific topics are addressed in individual collection requests under the generic clearance.
QDRL Staff use various techniques to evaluate interviewer administered, self-administered, telephone, Computer Assisted Personal Interviewing (CAPI) and Computer Assisted Self-Interviewing (CASI), Audio Computer-Assisted Self-Interview (ACASI), and web-based questionnaires including: cognitive interviewing, focus groups, usability testing, field tests/pilot interviews (personal/telephone), etc. All evaluations of questionnaires are recorded on audio or video tape unless there are clear reasons for not making recordings.
Cognitive interviewing offers a detailed depiction of meanings and processes used by respondents to answer questions—processes that ultimately produce the survey data. As such, the method offers an insight that can transform understanding of question validity and response error. The method is inexpensive and feasible; perhaps that is why it is one of the most used methods for pre-testing questions.
Respondents are not selected through a random process, but rather are selected for specific characteristics such as race or health status or some other attribute that is relevant to the type of questions being tested. Because the goal is to identify the presence of problems, as opposed to making estimations or causal statements, a randomly drawn sample is not required.
The interview structure consists of respondents first answering a draft survey question and then providing textual information to reveal the processes involved in answering the test question. Specifically, cognitive interview respondents are asked to describe how and why they answered the question as they did. Through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered.
In order to conduct analysis of these interviews, QDRL Staff have developed specifically designed analysis tools including Q-Video, Q-Notes, and Q-Bank.
Q-Video is a digitized video/audio application that captures, stores, and indexes the video and audio of a cognitive interview at the questionnaire level in a digitized database for the purpose of searching individual questions and conducting analysis.
Q-Notes is a software program that supports the structured collection and analysis of cognitive interview data. Hence, it serves as an audit trail tracing each finding to the original source. The video of the actual interview is embedded within the application so that the findings can truly be traced to the original source.
Q-Bank (a product of an interagency collaboration and hosted by NCHS), is a database consisting of evaluated questions from Federal surveys and links each question to the scientific report that evaluated the survey question. Questions are searchable by survey title, question topic (e.g. income, demographic, chronic health conditions), information type (e.g. objective characteristics, behavioral reports, attitudes), response category (e.g. yes/no, open-ended, quantity), response error (e.g. problems with terms, recall problems). In addition, users can search for keywords within individual questions. Q-Bank is intended to help users of survey data interpret the survey questions on which the data are based and understand the potential errors that might be associated with these questions.
Occasionally, focus groups (or group interviews of 5-10 individuals) are used to discuss general concepts that survey questions will focus on. Individual interviews are generally preferable to focus groups for evaluating specific questions because respondents usually respond to surveys individually, and the group dynamic can have a strong influence on interpretations and responses. However, focus groups can sometimes help questionnaire designers to understand the circumstances of various groups of people, and this information can be used to craft questions that better match respondent experiences.
Additional issues arise in computer-assisted survey instruments. Issues include the human-interface design, ease of use, comprehension, privacy, quality of on-line help and efficiency of screen organization. Optimal designs may be dependent on culture and education. Some of our research is designed to identify problems arising from the design of computer-based questionnaires.
Cognitive interviewing methodology identifies problems that are missed by traditional field tests. Field interviewers may not be sufficiently trained to identify questionnaire problems, and such tests are often conducted too late to allow for substantial revisions to be made. Nevertheless, field tests are a vital complement to cognitive interviews because they provide a better understanding of the magnitude of a problem. As time and resources allow, the behaviors of both interviewers and survey respondents in such interviews are observed and manually or audio recorded to allow for systematic analysis. These activities were used successfully to develop the questionnaires used in previous NHIS Supplements. The QDRL therefore plans to apply these techniques in development of the NHIS revised Periodic and Topical Modules (formerly referred to as Supplements) and of modules from other surveys.
Generally, field tests/pilot interviews for face-to-face surveys are conducted in the respondent's household, and pilot interviews for telephone surveys are conducted over the telephone. Professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) conduct these interviews. A subset of these interviews may be observed by a survey professional (NCHS staff member, or member of a Federal agency sponsoring the questionnaire). In cases involving observation, as the interviewer conducts the pilot interview, the observer compiles notes regarding respondent misunderstandings or difficulty answering, or questions that interviewers have difficulty administering, which help to identify potential question revisions. In addition, NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data. Subject matter staff are debriefed on these findings and the results of the field test/pilot interviews will be used to modify the questionnaire for follow-up field tests/pilot interviewing prior to the actual survey being conducted.
The practice of conducting field tests/pilot interviews allows testing of types of individuals who do not ordinarily volunteer for cognitive interviews in the laboratory; it also provides information collected under realistic field conditions, and collected early enough to be useful for questionnaire design decisions. This testing will be referred to in this document as field tests/pilot interviewing.
In addition to the applied questionnaire development activities described above, QDRL staff design and conduct research studies on the cognitive and interpretive aspects of survey methodology more generally. Such research could take the form of experiments embedded within fielded surveys (generally referred to as “split-ballot” experiments), experiments conducted in the laboratory, or exploratory studies employing individual interviews or focus groups. The purpose of the research is to enhance our understanding of the question response process, to develop better standards for questionnaire design, or to improve data collection procedures. Ultimately these studies produce generalizable knowledge that improves the quality of data collection instruments more generally.
All audio and video recordings are kept for as long as there is a justifiable use for the recordings as determined by the NCHS Research Ethics Review Board (ERB).
Items of Information to be Collected
This clearance request is for continuing the five types of activities that the QDRL carries out: 1) Survey questionnaire development and testing for CDC, other federal agencies, or other academic or professional institutions based on cognitive interviewing methodology; 2) Research on the cognitive and interpretive aspects of survey methodology; 3) Research on computer-user interface design for computer-assisted instruments e.g. Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self-Interviewing (CASI) instruments including Web-based surveys), also known as usability testing and Audio Computer-Assisted Self-Interview (ACASI); 4) Field tests/pilot interviews (either personal or telephone) are conducted with household respondents, using professional field interviewers; 5) Studies of the optimal design and presentation of statistical graphical and textual material. See 2.1 - 2.5 for a detailed explanation of these activities. Specific topics are addressed in individual collection requests under the generic clearance.
Information in Identifiable Form
Information in identifiable form (IIF) is collected for linkage of various QDRL forms (informed consent documentation, cash payment receipt form, and respondent demographics) and audiotape and videotape recordings. All of these items have been routinely approved and collected in the past. The identifiable information includes:
Name
Phone Number
Employment Status
Photographic Identifier (digital video image)
Identification of Website(s) and Website Content Directed at Children Under 13 Years of Age
The QDRL hosts no websites with content directed at children under 13 years of age.
2. Purpose and Use of Information Collection
The Questionnaire Design Research Laboratory (QDRL) conducts cognitive interviews, focus groups, field tests/pilot interviews, and experimental research in laboratory and field settings, both for applied questionnaire evaluation and more basic research on response errors in surveys.
The purpose and use of collecting this information fall into five categories:
2.1 Development and testing of specific survey questionnaires
2.2 Research on the cognitive and interpretive aspects of survey methodology
2.3 Research on human-computer interfaces/usability
2.4 Field tests/pilot interviewing
2.5 Studies of the optimal design and presentation of statistical graphical and textual material.
Development and cognitive testing of specific survey questionnaires:
The purpose of cognitive testing is not to obtain survey data, but rather to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the respondent, or otherwise contribute to the non-sampling errors of the survey.
Data collection procedures for cognitive interviewing are different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves: how respondents interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methods. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.
National Health Interview Survey (NHIS) (OMB # 0920-0214): The NHIS collects annual data on health status and limitations, use of health care, AIDS testing, family resources, health insurance, access to care, injury, health behaviors and functioning. Personal interviews are conducted in approximately 43,000 households including about 106,000 persons. The QDRL has conducted cognitive testing of various modules under 10-day packages since 1999, including mental health, alternative health, disability, insurance, strengths and difficulties services, cancer screening questions, complementary and alternative medicine, oral health, children’s mental health, voice, swallowing, speech and language, and sexual identity. In addition, the QDRL conducted a 50 case and a 575 case field test in conjunction with the NHIS. The 50 case test bridged early cognitive testing work conducted by the QDRL on the sexual identity questions with NHIS field interviews and focused on the programming of the ACASI portion of the questionnaire and the transition from orally administered to self-administered questions. The 575 case test employed field procedures consistent with the production NHIS, providing a more realistic reflection of the field effort required to obtain completed ACASI interviews and respondent acceptability of ACASI. It is anticipated that the QDRL will conduct cognitive testing for numerous modules as well as additional field tests in 2012-2015.
Other parts of the Department of Health and Human Services (DHHS), such as the Assistant Secretary for Planning and Evaluation (ASPE) occasionally request that NCHS include new policy relevant questions on the NHIS. Examples included citizenship status, health insurance coverage, and the receipt of government services by low-income individuals and families. The QDRL assisted in the development and pretesting of the survey questions. Assignments like this may occur during the 2012-2015 period and will be handled in a similar way. Further, in cases in which it may be difficult to identify and recruit the appropriate respondents (for example, persons who are undocumented aliens), contractors who have expertise in the use of cognitive techniques with difficult-to-locate populations will be enlisted to conduct the research with oversight by QDRL staff.
b) State and Local Area Integrated Telephone Survey (SLAITS) (OMB # 0920-0406): In response to the need for the development of a national capacity to generate high quality State and local level data for tracking and monitoring current and emerging health and welfare policy-related issues, the State and Local Area Integrated Telephone Survey (SLAITS) was initiated. SLAITS was designed to provide quick turnaround data on a variety of broad health and welfare related issues and includes questions on health insurance coverage, access to care, perceived health status, utilization of services, and measurement of child well-being. SLAITS uses the same Random-Digit-Dial (RDD) telephone design approach and sampling frame used in the National Immunization Survey (NIS).
The QDRL has conducted cognitive testing of various modules under 10-day packages since 1999, including children with special health care needs, asthma, and children’s health. It is anticipated that the QDRL will conduct testing for numerous modules during 2012-2015.
c) Pregnancy Risk Assessment Monitoring System (PRAMS): PRAMS is a surveillance project of the Centers for Disease Control and Prevention (CDC) and state health departments. PRAMS collects state-specific, population-based data on maternal attitudes and experiences prior to, during, and immediately following pregnancy.
The QDRL has conducted cognitive testing on PRAMS questionnaires under 10-day packages in 1999, 2001, 2003, and 2007. We anticipate testing new questions, as well as questions that will be proposed as expansions and refinements to those already found in PRAMS questionnaires during 2012-2015.
d) National Health and Nutrition Examination Survey (NHANES) (OMB # 0920-0237):
NHANES collects annual data about the health and diet of people in the United States. The survey consists of two parts: an in-home interview and a health examination. The in-home interview asks questions about health status, disease history, and diet. The health examination consists of tests based on age and gender and is performed in a Mobile Examination Center.
The QDRL has conducted cognitive testing of various modules under 10-day packages since 1999, including cognitive testing of a brochure designed to be used by the field interviewers to convert survey refusals, various modules for the “in-home interview,” including sexual orientation, physical activity and pain, positive prostate specific antigen (PSA), hypertension and pre-hypertension, audio-CASI sensitive questions, and creatine & life style questions. It is anticipated that the QDRL will conduct testing for numerous modules during 2012-2015.
e) National Immunization Survey (NIS) (Clearance exempt): The NIS is a list-assisted random-digit-dialing telephone survey followed by a mailed survey to children’s immunization providers that began data collection in April 1994 to monitor childhood immunization coverage.
The target population for the NIS is children between the ages of 19 and 35 months living in the United States at the time of the interview. Data from the NIS are used to produce timely estimates of vaccination coverage rates for all childhood vaccinations recommended by the Advisory Committee on Immunization Practices (ACIP). Estimates are produced for the nation and for each of 78 Immunization Action Plan (IAP) areas, consisting of the 50 states, the District of Columbia, and 27 large urban areas.
The QDRL has conducted cognitive testing of various modules under 10-day packages since 1999, including immunization, topical modules, knowledge, attitudes, and practices provider study, knowledge, attitudes, and practices household study. It is anticipated that the QDRL will conduct testing for numerous modules during 2012-2015.
f) National Survey of Family Growth (NSFG) (OMB # 0920-0314): The National Survey of Family Growth (NSFG) is a multipurpose survey based on personal interviews with a national sample of men and women 15-44 years of age in the civilian non-institutionalized population of the United States. Its main purpose is to provide reliable national data on marriage, divorce, contraception, infertility, and the health of adults and infants in the United States.
The QDRL has conducted cognitive testing of various NSFG modules under 10-day packages since 1999. It is anticipated that the QDRL will conduct testing for numerous modules during 2012-2015.
g) National Health Care Survey (NHCS) (various clearances):
The National Health Care Surveys are designed to answer key questions of interest to health care policy makers, public health professionals, and researchers. These can include the factors that influence the use of health care resources, the quality of health care, including safety, and disparities in health care services provided to population subgroups in the United States.
The QDRL has conducted cognitive testing of NHCS surveys and modules including NAMCS and NHAMCS Patient Record Evaluation Study; 2011 Physician Workflow Electronic Health Records (EMR) Supplement; 2012 Asthma Management Supplement; and the National Survey of Long-Term Care Providers. It is anticipated that the QDRL will conduct testing for numerous modules during 2012-2015.
h) Research on Perceptions of Quality of Life: QDRL staff will examine survey respondents’ perceptions of their self-assessed quality of life, and the basis for their responses to questions which purport to measure quality of life, especially from a health perspective. Such questions are increasingly important to both NCHS and CDC surveys as quality of life, rather then, simply length becomes a key measure. Questions from the Behavior Risk Factor Surveillance System (BRFSS) Quality of Life Module will be subject to ongoing evaluation by QDRL staff. In particular, cognitive testing will be conducted to determine whether modifications to question wording, response category ordering, and question re-ordering are likely to fundamentally affect the patterns of responses obtained. We also anticipate that several experts in the field of survey methodology and health assessment may be enlisted, under contracts, to assist in this research by, for example, conducting independent cognitive research, and comparing those results with those obtained in the NCHS Laboratory.
Other questionnaire testing and development: In addition to the specific questionnaire testing and development activities listed above, we anticipate that QDRL staff will be asked over the next three years to test questionnaires developed by NCHS, other components of CDC, other Federal agencies, and possibly academic and professional institutions that collect data relevant to public health. It is appropriate that the QDRL perform these activities, as it is currently only one of two Federal facilities performing cognitive interviewing in order to develop DHHS survey questionnaires, and is frequently the only one available for development of questionnaires originating outside of DHHS. However, because the requests may arrive with little advance notice, we cannot presently specify the nature of these questionnaires. Such a general plan was obtained in the previous clearance (No. 0920-0222), and the QDRL was thus able to conduct quick response testing of several questionnaires, including a Youth Traffic Safety Survey for the National Highway Traffic Safety Administration (NHTSA), Department of Transportation; testing and evaluation of Health Surveillance Maps for the Division of Adult and Community Health/the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP), CDC; testing and evaluation of the Revision of U.S. Standard Certificate of Live Birth for the Division of Vital Statistics, NCHS; testing and evaluation of International Disability questions for the Washington Group on Disability Statistics/UN Statistical Commission; testing and evaluation of competency questions believed to underlie both personal and societal success for the Program for the International Assessment of Adult Competencies, Center for Survey Research and Methodology ZUMA/Mannheim, Germany; testing and evaluation of a Respiratory Disease Questionnaire and a Lifetime Work History Questionnaire developed by the Division of Respiratory Disease Studies (DRDS), National Institute for Occupational Safety and Health (NIOSH),CDC; National HIV Behavioral Surveillance (NHBS) System for Division of HIV/AIDS Prevention (DHAP), National Center for HIV, Hepatitis, STD, and TB Prevention (NCHHSTP), CDC; Federal Statistical System (FSS) Public Opinion Survey. No procedural problems were encountered as a result of such testing. In fact, the flexibility associated with cognitive interviewing allowed for survey pre-testing in a timely manner that minimized respondent burden.
The interviews for questionnaire development activities (a) through (i) above will usually be conducted in QDRL facilities using cognitive interviewing procedures described in 2. If we are unable to obtain adequate numbers of individuals from particular population subgroups (e.g., elderly, or those who have specific health problems), we will attempt to make arrangements with organizations such as centers for the elderly, or service organizations for persons with specific health conditions, to interview respondents at outside locations.
Usually, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), or web-based. For a telephone interview, we will either make arrangements to call the respondent at home, or to conduct the interview in our laboratory, but calling the respondent from another laboratory room with face-to-face debriefing following.
It is possible that NCHS QDRL may collaborate with other agencies through interagency agreements which will include confidentiality provisions to test and develop survey questionnaires. Consent forms will be modified to reflect their participation.
2.2 Research on the Cognitive and Interpretive Aspects of Survey Methodology
The second major purpose of data collection is to conduct research on the cognitive and interpretive aspects of survey methodology:
a) Research on appropriateness of response scales: An important determinant of survey data quality is that questions include appropriate response scales. In particular, response scales must have clear meanings to respondents, and must allow them to adequately express their experiences. An emerging body of research suggests that seemingly trivial variations in response scales (e.g., using a scale from 1 to 10 as opposed to a scale from
-5 to +5) can significantly affect response distributions. Preliminary research has also been conducted on the meanings of vague quantifiers (such as often, sometimes, and rarely) and the benefits of certain scales over others (e.g., seven-point scales over feeling thermometers). QDRL staff will be engaged in additional research along these lines, possibly including cognitive laboratory testing of alternative response scales, as well as split-ballot experimentation.
b) Research on cognitive and interpretive aspects of nonresponse: Nonresponse creates numerous analytic difficulties on major surveys. Minimizing this problem requires a greater understanding of the cognitive processes that lead respondents to decide not to answer surveys or particular survey questions. QDRL staff plan to conduct cognitive interviews using a variety of types of survey questions (behavioral and attitudinal) in order to explore these decision processes further. Survey nonresponse will be explored through examination of reasons that nonresponders provide for their unwillingness or inability to complete surveys. It is also possible that data will be collected through experimental questionnaires administered outside of the laboratory that explore the effect of various design decisions on item nonresponse. Contracts may be used for some components of this data collection and analysis.
c) Respondent Perceptions of Confidentiality and Survey Participation: To encourage participation, NCHS surveys such as the NHIS and NHANES depend on advance letters, promising confidentiality and explaining uses of the data collected (a copy of the 2011 NHIS advance letter is included as Attachment C). However, it is not known how well these statements are generally understood, and believed, by survey respondents. Therefore, QDRL staff proposes to conduct cognitive interviews of laboratory respondents in order to examine their comprehension of such statements. The results will be used to propose modifications to procedures used to communicate key issues related to informed consent, and to explain the need and purpose for survey data in a way intended to increase survey participation.
d) General Methodological Research: QDRL staff constantly evaluate and refine the cognitive interviewing methods used at NCHS, especially in order to respond to changes such as the wide-spread introduction of CAPI (Computer Assisted Personal Interviewing) and Audio Computer-Assisted Self-Interview (ACASI) as a data-collection mechanism. Further, QDRL staff regularly conduct applied research on questionnaire design issues, such as the optimal wording for measures of complex concepts related to health status, utilization, and behavior. In 2012-2015 QDRL staff plan to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, the QDRL will work collaboratively with survey researchers from Universities and other Federal agencies to define and examine several research areas, including, but not limited to: 1) differences between face-to-face and telephone cognitive interviewing, 2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing, 3) reactions of both survey respondents and survey interviewers to the use of Computer Assisted Personal Interviewing (CAPI), Audio Computer-Assisted Self-Interview (ACASI) and 4) social, cultural and linguistic factors in the question response process. Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, questionnaires that are of current interest (such as NHIS Modules) may be evaluated using several of the techniques described above. Or, different versions of a survey question will be developed, and the variants then administered to separate groups of respondents in order to study the cognitive processes that account for the differences in responses obtained across different versions.
These studies will be conducted either by QDRL staff, DHHS staff, or NCHS contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.
2.3 Research on human-computer interfaces/usability
The third major purpose of this data collection is to conduct research on computer-user interface designs for computer-assisted instruments, which is often referred to as “usability testing.” This research examines how survey questions, instructions, and supplemental information are presented on computer instruments (e.g., CAPI or Computer Assisted Self-Interviewing (CASI) instruments), Audio Computer-Assisted Self-Interview (ACASI), and investigates how the presentation affects the ability of users to effectively utilize these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions that are not to be read to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on. Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task.
Usability testing has many obvious similarities to questionnaire-based cognitive research (described in Section 2.1), since it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat different, in that the typical user can be an interviewer (in the case of CAPI instruments) as well as a respondent (in the case of CASI/ACASI instruments). It also focuses more heavily on matters of formatting and presentation of information than traditional cognitive testing does.
2.4 Research Using Field Tests/Pilot Interviewing
The fourth major purpose of data collection is to apply unobtrusive field-based questionnaire evaluation techniques, especially with respect to future NHIS Topical and Periodic Modules and modules from other surveys. The different questionnaires may be pilot-tested either individually or in groups, depending on developmental status of the instruments, the appropriateness of combining them, and their overall length. For most field tests, professional field interviews (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) usually conduct approximately 200 pilot interviews (person/telephone). However, in 2012, a field test of almost 600 interviews was approved and larger sample sizes may be required for specific projects. There are four possible components to the proposed form of testing: a) a limited number of interviews on a draft version of the questionnaires are conducted using household respondents, b) a subset of interviews may be observed by NCHS and other staff trained in observational techniques, c) NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data and d) the potential for inclusion of built-in experiments i.e., two different versions of particular questions, in order to determine which version functions better in the field environment and mode of administration where test cases will be randomly assigned to receive questions in two different modes, i.e., CAPI or ACASI to assess the impact of mode on prevalence estimates and data quality.
2.5 Studies of the optimal design and presentation of graphical and textual material
The final major purpose is related to the growth of the Internet for collecting data (including Web-based surveys), and in disseminating health information. NCHS is the Federal government’s principal health statistics agency, and is responsible for collecting and disseminating many reports and volumes of data annually. During the last few years, the techniques developed for determining whether respondents understand survey questions have been applied with great utility to studying whether statistical publications and Web releases are optimally clear. One project, for example, involved the development and testing of a brochure designed by staff of the National Health and Nutrition Examination Survey (NHANES) to convert refusals to acceptance. Another recent project involved testing and evaluation of different Health Surveillance Map formats (choropleth versus isopleth) to determine if they affect ability to extract information from the maps for the Division of Adult and Community Health/the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP), CDC . We anticipate that there will be more work of this type during 2012-2015.
The major activities outlined above have well-demonstrated practical utility. As a result of laboratory testing, questionnaires may produce substantially less response error than would occur in the absence of this testing. Thus, users of NCHS data, in both Federal agencies and in the general health research community, will be less likely to be misled by erroneous statistical results. This assertion is supported by twenty four years of experience in using these techniques, and has been supported by findings presented at many statistical and research related conferences such as Joint Statistical Meetings (JSM), American Association for Public Opinion Research (AAPOR), American Sociological Association, and published in scientific journals such as Applied Cognitive Psychology, Journal of Official Statistics, Public Opinion Quarterly, Field Methods, and Quality and Quantity. The practical utility of field tests/pilot interviewing has also been supported in findings reported at an annual meeting of the American Statistical Association. Further evaluation of the efficacy of this method will be ongoing.
Privacy Impact Assessment Information
A Privacy Impact Assessment was submitted on October 29, 2011. The QDRL continues to collect, on a confidential basis, data needed in order to conduct QDRL studies. Data in identifiable form is collected for linkage of various QDRL forms (informed consent documentation, cash payment receipt form, and respondent demographics) and audio and videotape recordings. The QDRL also uses some identifiable data (name, phone number) to contact previous respondents for QDRL studies. The ability to match respondents to other data (informed consent documents, cash payment receipt forms, respondent demographics, and audio/videotapes) greatly expands the usefulness of the data at a very low cost.
Only those QDRL staff, specially designated agents as outlined in the informed consent form, and on occasion QDRL contractors, who must use the personal information for a specific purpose can use such data.
The collection of information in identifiable form requires strong measures to ensure that private information is not disclosed in a breach of confidentiality. All QDRL staff, as well as QDRL contractors receive appropriate training, and sign a “Nondisclosure Statement.” Staffs of collaborating agencies as outlined in the informed consent form are also required to become designated agents and sign the Nondisclosure Statement. Storage of confidential data is protected through procedures such as an internal QDRL LAN, passwords and carefully restricted access. See A10 for more details.
3. Use of Improved Information Technology and Burden Reduction
Usually, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), or web-based.
In addition, QDRL Staff use searches in Q-Bank to determine if a survey question has been cognitively tested. The regular use of Q-Bank reduces unnecessary testing as well as allows QDRL to build upon existing knowledge learned from past testing projects. Additionally, each cognitive interview is digitally recorded and stored on an internal, searchable video database. Like Q-Bank, this technology allows QDRL staff to build upon past projects and, at the same time, it improves the accountability of test findings.
4. Efforts to Identify Duplication and Use of Similar Information
The QDRL at NCHS is the only government facility that currently conducts testing and development of NCHS or other CDC questionnaires, and is frequently the only one available to test questionnaires from other agencies. Similar facilities at the Bureau of the Census and the Bureau of Labor Statistics bear the responsibility for testing survey questionnaires associated with their own agencies. The demand for QDRL activities exceeds available resources.
It is also possible that the NCHS QDRL may collaborate through interagency agreements which will include confidentiality provisions to test and develop survey questionnaires. Consent forms will be modified to reflect their participation. This work, however, is not duplicative, but cooperative in nature, and should result in a higher quality final product. Researchers in the NCHS Questionnaire Design Research Laboratory also maintain very close contact with other experts in the field of questionnaire development in the academic survey community, in the health sciences field, at the Bureau of the Census, the Bureau of Labor Statistics, General Accounting Office, the National Science Foundation, National Cancer Institute, and the Energy Information Administration. From these contacts, it is clear that no other projects that duplicate the current proposal are now underway.
5. Impact on Small Businesses and Other Small Entities
In the past, representatives of small businesses have been interviewed as part of testing of establishment surveys, such as the National Employer Health Insurance Survey (NEHIS) (OMB# 0920-0341). If such requests are made, these businesses will be approached in the same manner as the individuals we normally recruit; we will ask the organization to identify the appropriate staff members with whom to conduct the cognitive interviews.
6. Consequences of Collecting the Information Less Frequently
Individual projects usually involve one-time data collection activities. There are no legal obstacles to reducing the burden.
7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5
There are no special circumstances.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies
8.1 A Federal Register notice for this collection was published on February 24, 2012, (Vol. 77, No. 37, p. 11124-11125). The text of the notice is contained in Attachment B1. One non-substantive comment was received to date (Attachment B2). CDC sent a general response.
8.2.1 Other agencies: some of the topics selected for the NHIS are anticipated
to be requested by other agencies. These agencies have been involved in
development of survey objectives and the draft questionnaires.
8.2.2 Other individuals: Researchers who have special interest and expertise in the research areas explored will be contacted as necessary.
Consultants outside of CDC:
Gordon Willis
Cognitive Psychologist
Applied Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
6130 Executive Blvd, MSC 7344
EPN 4005
Bethesda, MD 20892-7344
Alisu Schua-Glusberg
Research Support Services
906 Ridge Avenue
Evanston, IL 60602
(847) 864-5677
Floyd J. Fowler, Jr., Ph.D.
Center for Survey Research
University of Massachusetts Boston
100 Morrissey Blvd.
Boston, Massachusetts 02125
(617) 287-7200
Consultants within CDC:
Jane Gentleman, Ph.D.
Director, Division of Health Interview Statistics
Room 2218
National Center for Health Statistics
3311 Toledo Rd.
Hyattsville, MD 20782
(301) 458-4001
Stephanie Ventura
Division Vital Statistics
Room 7418
National Center for Health Statistics
3311 Toledo Rd.
Hyattsville, MD 20782
(301) 458-4547
b) There are no unresolved problems.
c) Consultation with representatives of those from whom data will be collected will take place in the form of interviews with volunteers to determine the feasibility of collecting the needed data, the most promising approach for data collection, and general attitudes about the respondents which might influence data collection.
9. Explanation of Any Payment or Gift to Respondents
For most testing projects, cognitive interview respondents receive remuneration for several reasons:
Typically, respondents are recruited for specific characteristics that are related to the subject matter of the survey (e.g., questions may be relevant only to people with certain health conditions). The more specific the subject matter, the more difficult it is to recruit eligible respondents. Remuneration helps to attract a greater number of potential respondents.
Cognitive interviews require an unusual level of mental effort, as respondents are asked to explain their mental processes as they hear the question, discuss its meaning and any ambiguities, and describe why they answered the questions the way they did.
They are usually asked to travel to the laboratory testing site, which involves transportation and parking expenses. (Many respondents incur additional expenses due to leaving their jobs during business hours, making arrangements for child care, etc.).
For a standard cognitive interviewing project in which one-hour interviews are conducted at NCHS and eligibility requirements are of average complexity, respondents will be given $40.00. The remuneration may be reduced to an amount no lower than $30.00 if the interview is of shorter duration, or does not require the respondent to travel to NCHS. Higher remunerations may be requested on a case-by-case basis for particularly difficult recruitments. For example, in a 2008 & 2009 study, the QDRL was unable to find epidemiologists willing to be interviewed for less than $75 and in 2011 the QDRL was unable to find physicians willing to be interviewed for less than $100. On rare occasions, a lower remuneration is proposed.
It is important to offer remuneration sufficient to attract the full range of needed respondent types for cognitive interviewing projects. Inadequate respondent recruitment limits the effectiveness of the questionnaire evaluation. Requests and justification for remuneration will be included in each individual collection submission.
For activities that are meant to resemble the usual household interview (for example, field tests/pilot interviewing), respondents will not receive remuneration.
10. Assurances of Confidentiality Provided to Respondents
Confidentiality provided to respondents is assured by adherence to Section 308(d) of the Public Health Service Act (42 USC 242m) which states:
"No information, if an establishment or person supplying the information or described in it is identifiable, obtained in the course of activities undertaken or supported under section...306 (NCHS legislation),...may be used for any purpose other than the purpose for which it was supplied unless such establishment or person has consented (as determined under regulations of the Secretary) to its use for such other purpose and (1) in the case of information obtained in the course of health statistical or epidemiological activities under section...306, such information may not be published or released in other form if the particular establishment or person supplying the information or described in it is identifiable unless such establishment or person has consented (as determined under regulations of the Secretary) to its publication or release in other form,..."
In addition, legislation covering confidentiality is provided according to section 513 of the Confidential Information Protection and Statistical Efficiency Act (PL 107-347) which states:
“Whoever, being an officer, employee, or agent of an agency acquiring information for exclusively statistical purposes, having taken and subscribed the oath of office, or having sworn to observe the limitations imposed by section 512, comes into possession of such information by reason of his or her being an officer, employee, or agent and, knowing that the disclosure of the specific information is prohibited under the provisions of this title, willfully discloses the information in any manner to a person or agency not entitled to receive it, shall be guilty of a class E felony and imprisoned for not more than 5 years, or fined not more than $250,000, or both.”
Privacy Impact Assessment Information
The NCHS Privacy Act Coordinator has reviewed this request and has determined that the Privacy Act is applicable. The System of Records name is Health and Demographic Surveys Conducted in Probability Samples of the U.S. Population. The Privacy Act System of Records number for the QDRL is 09-20-0164.
Data will be treated in a confidential manner. The process of informing respondents of the procedures used to keep information confidential begins with the telephone screener and will carry through to the interviewer and all communications with potential respondents. Materials will include all elements of informed consent, including the purpose of the data collection, the voluntary nature of the study, audio or videotaping of the interview, and the effect upon the respondent for terminating the interview at any time.
Informed Consent and Voluntary Nature
QDRL respondents/interviews conducted at NCHS
QDRL respondents are usually recruited by expressing their personal willingness to participate. They read or hear about the study through media advertisements, flyers, and word-of-mouth, and either call the QDRL answering machine number or contact a person coordinating the recruitment. Thus, participation is strictly voluntary and respondents are not chosen randomly. Data collection for this project is authorized under 42 USC 242k (Section 306 of the Public Health Service Act).
During the telephone screener, potential respondents are informed that answering the telephone screener questions to determine their eligibility for the study is completely voluntary. They are informed that we are required by law to use the information they provided in the telephone screener for statistical research only and to keep it confidential, and that the law prohibits us from giving anyone any information that may identify them without their consent. In addition, respondents who are determined to be eligible for the study are informed during the telephone screener that the information they provide during the cognitive interview is confidential.
Prior to the start of the cognitive interview, QDRL respondents read and sign Attachment D, Informed Consent Form (written at an 8th grade reading level). There are five templates in the attachment to cover various consent situations. The consent form states that participation is voluntary, they are free to terminate the interview at any time, and if they do so, they will still receive remuneration. The consent form describes the purpose of the interview, taping, specifies that the tapes may be played for other staff working closely on that project, that voice and face identifiers will remain on the tapes, and that they may be recognized by a staff member viewing or listening to the tapes. Respondents are given a copy of the consent form, which contains contact information for the QDRL Laboratory Manager, the NCHS Research Ethics Review Board (ERB), and the NCHS Confidentiality Officer.
At the close of the cognitive interview, a respondent may also be asked by the interviewer to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings Form. The purpose of this form is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. Respondents are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer. If respondents grant Special Consent, recordings are kept for as long as there is a justifiable use for the recordings as determined by the NCHS Research Ethics Review Board.
Additionally, at the close of the interview the respondent may also be asked by the interviewer to sign Attachment I, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL respondents’ privacy and the recording until it is returned to NCHS. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interview with minors will never be shown to others not included in the study staff. Respondents are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer.
QDRL respondents/interviews conducted off-site1: Sometimes interviewers must travel to establishments or individuals’ homes in order to conduct interviews when it is not feasible for respondents to travel to the QDRL. It is the interviewer’s responsibility to take necessary steps to ensure confidentiality and safeguarding of materials. Interviews at establishments should be conducted in private rooms with a closed door. If no private room is available, the respondent can select a private area and the interviewer will judge whether the area is private enough to ensure confidentiality. If the interviewer assesses that the area is not private and/or soundproof enough, and no alternative area can be provided, the interview is canceled. For those surveys conducted in the respondent’s home, the interviewer requests in advance that the respondent arrange for privacy. However, interview location within the home is the choice of the respondent. In all cases, extreme care is taken with audio and video recordings and any materials that contain personal identifiers such as the Cash Payment Receipt form, the Informed Consent Form, Special Consent for Expanded Use of Video and Audio Recordings, or the Special Consent to Send Video and Audio Recordings to Off-site Researchers. Materials are then transported to the QDRL, where standard procedures are followed.
Focus groups: In focus group settings, respondents are together and obviously can hear each other’s comments, statements, and questions. Respondents are told in their initial telephone screening interview that they will be participating in a discussion group with other volunteers. Before the group discussion begins, respondents sign the Informed Consent Form which is tailored to specify that they will be participating in a focus group. The Informed Consent also states that they will be asked to pick a name and put it on a name tag, and that they do not have to use their real name. It is the responsibility of the interviewer (usually referred to as a moderator when conducting a focus group) to instruct the group that the information discussed will be held confidential by NCHS staff and should be treated confidentially by all respondents. Respondents are strongly urged to respect the privacy of the other respondents and not to discuss with others what was discussed by the group.
Contractor conducted interviews
On the rare occasion when contractors are used to collect data as part of QDRL projects, they are contractually bound by NCHS confidentiality provisions and must submit documentation concerning their safeguarding practices to NCHS prior to data collection. This is standard NCHS practice and does not reflect a special QDRL procedure. The contractor employee will view the NCHS Confidentiality video and sign the NCHS non-disclosure statement before starting work on the project.
Field Tests/Pilot Interviews
For field test/pilot interviews of household and telephone respondents, standard operating procedures regarding informed consent and survey administration procedures specific to the survey being tested will be followed.
Security
Confidentiality of responses and safeguarding of data at NCHS
The QDRL has a routine set of measures to safeguard confidentiality, including the following: all QDRL staff who have access to confidential information are given instruction by the QDRL Laboratory Manager on the requirement to protect confidentiality and are required to sign a pledge to maintain confidentiality; only such authorized QDRL personnel are allowed access to confidential records and only when their work requires it; when cognitive interviews or focus groups are conducted off-site, data are secured to insure that there is no loss in transit; and when confidential information is not in use, it is stored in secure conditions.
All respondents receive a copy of Attachment D, Informed Consent Form, which describes the procedures by which confidentiality of data identifying individuals is maintained.
Informed Consent documents
Informed consent documents are stored by project in a separate drawer in a locked filing cabinet in the locked office of the QDRL Laboratory Manager. No one other than the QDRL Manager has access to the informed consent forms.
Cash Payment Receipt form
Cash Payment Receipt forms are stored by project in a separate drawer in a locked filing cabinet in the locked office of the QDRL Laboratory Manager. No one other than the QDRL Manager has access to the informed consent forms. The Cash Payment Receipt forms are kept for accounting and auditing purposes.
QDRL Respondent Database: A custom-designed QDRL Respondent Database contains personal identifiable information and demographic information on respondents who have participated in past QDRL studies such as name, phone number, age, marital status, ethnicity, race, education, employment status, and household income. The QDRL Respondent Database is used to conduct computerized searches to locate computer records of past respondents having salient characteristics for use in future studies. The QDRL Respondent Database is also used to produce periodic tabulations and reports on database characteristics and response rates. The computer containing the QDRL Database is password protected and located in the QDRL Laboratory Manager’s locked office.
Safeguarding of audio and video recordings: The QDRL Laboratory Manager/QDRL Staff label each audio and video recording by a unique respondent identifier number, date, time, and project title. No other identifying information is labeled on the recording. Recordings are viewed/listened to in the Observation Room, or on QDRL Staff’s desk top computers which are hardwired to the secure QDRL Local Area Network (LAN). Only QDRL Staff holding proper passwords and keys to staff locked offices have access to interview recordings on the QDRL LAN. The QDRL LAN is located in the QDRL Control Room and locked when not in use. Because the server is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS. Recordings are stored in the Observation Room which is locked when not in use, or on the secure QDRL LAN.
Safeguarding of video recordings viewed at locations other than NCHS: Depending on the project, sponsors and collaborators may be from CDC, and occasionally from other DHHS or outside Federal agencies. The Informed Consent Form is tailored to describe each project and will specify which agencies are collaborating in the research and which staff(s) may be viewing the recording. Any outside NCHS collaborator viewing the recording (whether onsite at NCHS or off-site at a collaborating agency) will be required to sign a Nondisclosure Affidavit. Occasionally, a collaborator will be unable to travel to NCHS to view the recording. For those recordings in which Special Consent to Send Video and Audio Recordings to Off-site Researchers were granted, a contract will be developed and signed by the Director of NCHS and the Director of the relevant organization. The contract will state how they will protect the respondents’ privacy and the recording until its return to NCHS. The contract will be coordinated through the NCHS Confidentiality Officer who will oversee shipment and the return of the recording to NCHS.
Reports and publications: No respondent names or other identifying information is included in any reports or publications of cognitive testing results.
Presentations: No respondent names or other non-photographic identifying information is included in any presentations of cognitive testing results. As outlined in the standard informed consent and the special consent for expanded use of video and audio recordings, QDRL respondents have been informed that voice and face identifiers will remain on the tape and have granted permission for the audio or videotape recordings to be played either to individuals working closely on the project or at conferences, meetings, or in the classroom.
Protocol #2010-19 Laboratory Based Questionnaire Design (QDRL) was approved by the NCHS Research Ethics Review Board on October 13, 2011 (Attachment J).
11. Justification for Sensitive Questions
Most of the questionnaires currently proposed for study generally do not contain questions that are highly sensitive in nature. There are some exceptions, such as the National Survey of Family Growth, NHIS questions on income, HIV and sexual identity, and NHANES questions on sexual behavior. Again, one purpose of pre-testing such questions is to determine means for fashioning these questions in such a way that sensitivity is minimized, and responses are valid. Attachment J contains NCHS Research Ethics Review Board approval for research involving human respondents. Protocol #2010-19 Laboratory Based Questionnaire Design (QDRL) was approved by the NCHS Research Ethics Review Board on October 13, 2011 (Attachment J).
12. Estimates of Annualized Burden hours and costs:
An average of 3,100 individuals (9,300 over the three year period) participate in QDRL activities in a given year and the average annual respondent burden is estimated to be 3150 hours. Since this is a generic clearance, the burden table shows the total burden for the full three years, i.e. 9,450 hours. Estimates of respondent burden for each of the questionnaire development studies, over the course of data collection, are provided below. Estimates are based mainly on the practice of conducting one-hour interviews with respondents. The estimates cover the time that each respondent will spend communicating with the Laboratory Manager (see Attachment K), in answering screener questions and survey questions and, in some cases, being debriefed about the decision and recall strategies they used. For field tests/pilot interviews it is anticipated that interviews will last one hour. For our General Methodological Research studies, questionnaire administration is anticipated to frequently require less than an hour of a respondent’s time (for example, a fifteen-minute interview may be conducted), and in rare cases, the burden may be more than one hour (although not more than 2 hours). Because the hours per response are expected to vary, we will select the final sample size for each project in such a way that the total burden hours do not exceed the estimate listed above. For focus groups, the usual amount of time is 90 minutes (1.5 hours) with instructions and ancillary paperwork processes taking an additional 15-25 minutes.
For interviews in the laboratory, time required to travel to the lab is not covered, because distances and modes of transportation are unknown. No retrieval of information by respondents is anticipated; although it is possible that validation of data at some point may require respondents to check records, probably those kept at home. In that case, the study will be designed so that the response time includes record retrieval. All estimates are based on NCHS' experience with 1988 through 2011 questionnaire development activities.
Estimated Burden Table
Projects |
Number of Respondents |
Number of Responses/ Respondent |
Average hours per response |
Response burden
|
QDRL Interviews |
9000 |
1 |
1 |
9000 |
Focus groups |
300 |
1 |
1.5 |
450 |
Total |
|
|
|
9450 |
B. Annualized costs to respondents.
No costs are anticipated. Remuneration to respondents are designed to compensate them for their effort and any out-of-pocket costs.
13. Estimates of Other Total Annual Cost Burden to Respondents and Recordkeepers
None.
14. Annualized Costs to the Federal Government
The cost to the government consists mainly of the salaries of the QDRL staff that will (1) assist the questionnaire designers in the design of appropriate laboratory instruments, (2) recruit, schedule, and assist in interviewing volunteer respondents, and (3) assist in the analysis of the results and recommend changes in questionnaire wording.
Total annualized project costs are as follows:
NCHS costs for QDRL staff to plan, conduct, and analyze the outcomes of the questionnaire development activities:
Managerial 1.00 FTE $116,000.00
Professional 7.00 FTE $506,000.00
Support 1.00 FTE $89,000.00
Remuneration of QDRL respondents 3100 @ $40 $124,000.00
QDRL Contract Staff (including remuneration) $630,000.00
Contracts for assistance with
methodological research $30,000.00
Off-site travel: (see note below under
travel costs) $10,000.00
Materials for conducting household
interviews $500.00
Flyers $200.00
Advertisements $16,000.00
Hardware and software upgrades $50,000.00
Annual Total $1,571,700
3 Year Total (for generic submission) $4,715,100.00
Travel costs: Most data will be collected in NCHS office space. However, it will be more efficient in certain instances to hold interviews with individuals at other locations, which will involve some travel costs. Further, household interviews will require limited numbers of in-person interviews in respondent households. Household interviews will be done locally, in order to limit travel costs, unless there is a compelling reason to do otherwise (for example, if respondents critical to the study can be interviewed only at a distant location).
15. Explanation for Program Changes or Adjustments
This is a generic clearance. Current burden is 1800 hours. We are requesting 9450 burden hours, which is a difference of 7650 hours. The difference is due to an increase in the number and size of projects being undertaken.
16. Plans for Tabulation and Publication and Project Time Schedule
This clearance request is for questionnaire development activities to be conducted prior to survey production and for developmental work that will guide future questionnaire design. The majority of laboratory investigations will be analyzed qualitatively. The survey designers and lab staff serve as interviewers and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. Final reports will be written that document how the question performed in the interviews, including question problems as well as the phenomena captured by the survey question. All reports will be placed on Q-Bank for public access. Reports are used to provide necessary information to guide designs for redesigning a question prior to fielding as well as to assist end users when analyzing the survey data. For field tests/pilot interviewing activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews in order to determine where additional problems occur. Because NCHS is using state-of-the-art questionnaire development techniques, methodological papers will be written which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the cognitive interview and observation techniques.
17. Reason(s) Display of OMB Expiration Date is Inappropriate
Not applicable.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
Not applicable.
1Off-site interviews fall into two categories. First, it is not always feasible for individuals to travel to the QDRL, or it may be more efficient for interviewers to travel to a particular site. Second, we occasionally conduct establishment studies where a visit to the business location is pertinent to data collection.
| File Type | application/msword |
| File Title | Supporting Statement for Request for Clearance: |
| Author | Karen Whitaker |
| Last Modified By | Moien, Mary (CDC/OSELS/NCHS) |
| File Modified | 2012-03-23 |
| File Created | 2012-03-23 |