Ssa_0222_07292021
SSA_0222_07292021.docx
Collaborating Center for Questionnaire Design and Evaluation Research
OMB: 0920-0222
Supporting Statement A for Request for Clearance:
COLLABORATING CENTER FOR QUESTIONNAIRE DESIGN AND EVALUATION RESEARCH
OMB No. 0920-0222
Expiration Date: 08/31/2021
Contact Information:
Amanda Titus B.S.
Behavioral Scientist, Collaborating Center for Questionnaire Design and Evaluation Research
Division of Research and Methodology
National Center for Health Statistics/CDC
3311 Toledo Road, Room 5451
Hyattsville, MD 20782
301-458-4579
July 29, 2021
Table of Contents
A. Justification
A.1. Circumstance Making the Collection of Information Necessary 5
A.2. Purpose and Use of Information Collection 10
A.3. Use of Improved Information Technology and Burden Reduction 20
A.4. Efforts to Identify Duplication and Use of Similar Information 20
A.5. Impact on Small Businesses or Other Small Entities 20
A.6. Consequences of Collecting the Information Less Frequently 20
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 21
A.8. Comments in Response to the Federal Register Notice and
Efforts to Consult Outside the Agency 21
A.9. Explanation of Any Payment or Gift to Respondents 22
A.10. Protection of the Privacy and Confidentiality of Information Provided
by Respondents…………………………………………….. 23
A.11. Institutional Review Board (IRB) and Justification for Sensitive Questions 30
A.12. Estimates of Annualized Burden Hours and Costs 34
A.13. Estimates of Other Total Annual Cost Burden to Respondents
or Record Keepers 36
A.14. Annualized Cost to the Federal Government 36
A.15. Explanation for Program Changes or Adjustments 37
A.16. Plans for Tabulation and Publication and Project Time Schedule 37
A.17. Reason(s) Display of OMB Expiration Date in Inappropriate 37
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions 37
LIST OF ATTACHMENTS
ATTACHMENT A: Public Health Service Act
ATTACHMENT B: 60-day Federal Register Notice
ATTACHMENT B1: Public Comment
ATTACHMENT C: Nondisclosure affidavit
Template 1: Federal Employee Version
Template 2: Contractor Version
ATTACHMENT D: Sample screening script
Template 1: Respondent recruited from newspaper
advertisement
Template 2: Respondent recruited from CCQDER Database
ATTACHMENT E: Informed consent templates for CCQDER interviews
Template 1: Adults
Template 2: Adults offsite
Template 3: Focus Groups
Template 4: Virtual
Template 5: Parental/Guardian
ATTACHMENT F: Special Consent for Expanded Use of Video and Audio Recordings
ATTACHMENT G: Special Consent for Expanded Use of Video and Audio Recordings
for Individual Respondents of Discussion Groups
ATTACHMENT H: IRB Approval Documentation (NCHS Research Ethics Review Board)
H-1: NCHS ERB Approval
ATTACHMENT I: Sample recruitment flyer
ATTACHMENT J: Sample newspaper advertisement
ATTACHMENT K: Sample CCQDER Voice mail script
ATTACHMENT L: Sample of a 2017 advance letter sent to NHIS respondents
ATTACHMENT M: Sample Respondent data collection sheet
ATTACHMENT N: Detailed explanation of cognitive interviewing procedure read by interviewer to respondent
ATTACHMENT O: Thank You Letter
Goal
of the study: It
is the goal of the Collaborating Center for Questionnaire Design
and Evaluation Research (CCQDER) to not only evaluate questions
for optimal design, but also to provide documentation supporting
the validity of NCHS survey data.
Intended
use of the resulting data:
CCQDER obtains information about the processes people use to
answer survey questions as well as to identify any potential
problems in the questions, e.g., questions which are vague or
ambiguous, cannot be answered readily or accurately by the
respondent, or otherwise contribute to the non-sampling errors of
the survey.
Methods
to be used to collect:
CCQDER Staff use various techniques to design and evaluate
interviewer administered, self-administered, telephone, Computer
Assisted Personal Interviewing (CAPI) and Computer Assisted
Self-Interviewing (CASI), Audio Computer-Assisted Self-Interview
(ACASI), and web-based questionnaires. These methods include
cognitive interviewing, focus groups, usability testing,
ethnography, and field tests/pilot interviews
(personal/telephone/web).
Subpopulation
to be studied:
For qualitative methods, respondents are not selected through a
random process, but rather are selected for specific
characteristics such as race or health status or some other
attribute that is relevant to the type of questions being tested.
How
data will be analyzed:
The goal of methods such as cognitive interviewing is to identify
the presence of interpretive patterns as opposed to making
estimations or causal statements. Depending on the needs of
the project, a variety of analysis methods may be used (such as
statistical analysis of non-response) and will be explained in the
individual information collection requests under this clearance.
Supporting Statement A
Collaborating Center for Questionnaire Design and Evaluation Research
A three-year OMB clearance revision is requested for “NCHS Collaborating Center for Questionnaire Design and Evaluation Research” (currently approved under the title of “NCHS Questionnaire Design Research Laboratory”) (OMB No. 0920-0222, Exp. Date 08/31/2021). This generic clearance request encompasses general questionnaire development, pre-testing, measurement-error reduction activities to be carried out in 2021-2024 in the Division of Research and Methodology, National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC). The activities are to be conducted by the staff of the CCQDER Collaborating Center for Questionnaire Design and Evaluation Research and designated agents such as contractors, if needed and involve the development of health-related survey questionnaires, using a methodology which has been employed effectively since 1985. This revision seeks only an increase in burden hours to allow the CCQDER to conduct more studies. Otherwise there are no substantive changes to the package.
A. JUSTIFICATION
1. Circumstances Making the Collection of Information Necessary
In 1983/1984, the Committee on National Statistics conducted a two-part seminar on the Cognitive Aspects of Survey Methodology (CASM) under a grant from the National Science Foundation (NSF). The respondents in the CASM seminar (CASM I) were survey researchers and cognitive psychologists from academic institutions and survey researchers from the National Center for Health Statistics (NCHS) and the Bureau of the Census. The seminar examined a number of cognitive-related methodological studies that might lead to improvements in the questionnaires and interviewing procedures employed in scientific surveys in general, and in the National Health Interview Survey (NHIS) as a test case.
Following this seminar, the NSF provided funding to NCHS to investigate how relevant knowledge and techniques in cognitive science could be applied to improve health surveys. The project, begun in 1984, was called Laboratory-Based Studies of the Cognitive Aspects of Survey Methodology (CASM), and used cognitive psychological methods to study the survey interviewing process. In its final report, NCHS concluded that it is feasible and efficient for a Federal statistical agency to conduct laboratory research on the cognitive aspects of survey questionnaires. Subsequently, NCHS applied the cognitive research techniques being tested under the grant to develop the 1987 NHIS supplement (OMB No. 0920-0214, Exp. Date 12/31/2023, a comprehensive set of questions on knowledge, attitudes, and practices regarding cancer risk factors. Cognitive research techniques (also known as cognitive interviewing) proved invaluable for identifying conceptual problems with draft questions. The NCHS project staff concluded from this experience that past questionnaire design procedures were often unable to identify questions that were failing to measure what was intended, but that interviews in the laboratory were effective for identifying these kinds of measurement errors. The Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) was created at NCHS to provide such testing for NCHS surveys on a regular basis, as well as to continue more general research on the survey response process, questionnaire design, and pretesting methodology.
In October 2009, NCHS held a Question Evaluation Methods Workshop to examine various question evaluation methods as well as to discuss the impact of question design and evaluation on survey data quality. Broad consensus determined that measurement error in the Federal statistical enterprise requires renewed consideration. Federal statistical agencies have a fundamental obligation to produce valid and reliable data, and more attention needs to be placed on question evaluation and documentation. Furthermore, it was established that validation of measures is a particularly complex, methodological problem that requires a mixed-method approach. While quantitative methods are essential for understanding the magnitude and prevalence of error, they remain dependent on the interpretive power of cognitive interviewing. Unlike any other question evaluation method, cognitive interviewing can portray the interpretive processes that ultimately produce survey data. As it is practiced as a qualitative methodology, cognitive interviewing reveals these processes as well as the type of information that is transported through statistics. Consequently, cognitive interviewing is an integral method for ensuring the validity of statistical data, and the documented findings from these studies represent tangible evidence of how the question performs.
It is the mission of the CCQDER to not only evaluate questions for optimal design, but also to provide documentation supporting the validity of NCHS survey data. Specifically, the final product of each study conducted under the auspices of this clearance will be a clearly documented and publicly available report. Such documentation also serves NCHS data users, allowing them to be critical users in their approach and application of the data. Consequently, all completed CCQDER testing reports are located and made accessible on Q-Bank (http://wwwn.cdc.gov/qbank), an interagency, online searchable database that houses question evaluation studies.
Data collection for this project is authorized under 42 U.S.C. 242k (Section 306 of the Public Health Service Act). A copy of the legislation is provided in Attachment A. CDC is requesting terms of clearance identical to previous submissions. CDC will submit individual collections under this generic three year clearance to OMB. It is requested that OMB continue to provide feedback on the individual collections within 10 working days of the submission. Standard incentives for cognitive interviewing respondents will be capped at $40.00 for an hour interview, though higher incentives may be requested with justification for difficult recruitments, such as medical doctors and practitioners, specialized populations, focus groups, etc.
Overview of the Data Collection System
CCQDER Staff are methodological specialists who examine questionnaires from NCHS, CDC, other federal agencies, or other academic or professional institutions. Specific topics are addressed in individual collection requests under this generic clearance.
CCQDER Staff use various techniques to design and evaluate interviewer administered, self-administered, telephone, Computer Assisted Personal Interviewing (CAPI) and Computer Assisted Self-Interviewing (CASI), Audio Computer-Assisted Self-Interview (ACASI), and web-based questionnaires. These methods include cognitive interviewing, focus groups, usability testing, ethnography, and field tests/pilot interviews (personal/telephone/web). Unless there are clear reasons for not making recordings, most evaluations of questionnaires will be video and audio recorded to allow researchers to review not only the transcript of the interaction, but also the behaviors and body language of the respondents. Interviews conducted offsite—outside of the physical Collaborating Center for Questionnaire Design and Evaluation Research at NCHS— may be audio or video recorded. In general, these recordings will allow researchers to insure the quality of their interview notes.
For qualitative methods, respondents are not selected through a random process, but rather are selected for specific characteristics such as race or health status or some other attribute that is relevant to the type of questions being tested. Because the goal of methods such as cognitive interviewing is to identify the presence of interpretive patterns as opposed to making estimations or causal statements, a purposive sample rather than one randomly drawn is utilized. However, for quantitative field tests, a wider range of sampling strategies will be employed—depending on the specific purpose of the test. Examples of these strategies might include Random Digit Dialing (RDD) and web-based panels designed to match certain demographics.
To rigorously and systematically examine the cognitive interviewing, focus group, and usability testing data, online and internal applications have been developed. They include: Q-Video, Q-Notes, Q-Notes Plus and Q-Bank.
Q-Video is a digitized video/audio application that captures, stores, and indexes the video and audio of a cognitive interview at the questionnaire level in a digitized database for the purpose of searching individual questions and conducting analysis.
Q-Notes is a qualitative research application developed to assist in the management of and allow for more rapid, yet thorough systematic collection and analysis of cognitive interviewing studies. Q-Notes provides interviewers and analysts real-time access to interview data and allows interviews to be conducted in multiple geographical regions so that comparability can be examined for multi-national and multi-lingual surveys. Q-Notes is currently used by various academics and statistical agencies around the world both for their own projects and to collaborate with other agencies internationally.
Q-Notes Plus is an extension of Q-Notes, where the actual interview is embedded within the application so that the findings can truly be traced to the original source.
Q-Bank (a product of an interagency collaboration and hosted by NCHS) is a database consisting of evaluated questions from Federal surveys and links each question to the scientific report that evaluated the survey question. Questions are searchable by survey title, question topic (e.g. income, demographic, chronic health conditions), information type (e.g. objective characteristics, behavioral reports, attitudes), response category (e.g. yes/no, open-ended, quantity), and response error (e.g. problems with terms, recall problems). In addition, users can search for keywords within individual questions. Q-Bank is intended to help users of survey data interpret the survey questions on which the data are based and understand the potential errors that might be associated with these questions.
The most commonly used method under this general clearance will be cognitive interviewing. Cognitive interviews offer detailed depictions of meanings and processes used by respondents to answer questions—processes that ultimately produce the survey data. As such, the method offers insights that can transform the understanding of question validity and response error.
The interview structure tends to consist of respondents first answering a draft survey question and then providing textual information to reveal the processes involved in answering the test question. Specifically, cognitive interview respondents are asked to describe how and why they answered the question as they did. Through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered.
Occasionally, focus groups (or group interviews of 5-10 individuals) are used to discuss general concepts that survey questions will focus on. Individual interviews are generally preferable to focus groups for evaluating specific questions because respondents usually respond to surveys individually, and the group dynamic can have a strong influence on interpretations and responses. However, focus groups can sometimes help questionnaire designers to understand the circumstances of various groups of people, and this information can be used to craft questions or methodologies that better match respondent experiences.
Additional issues arise in computer-assisted survey instruments. Issues include the human-interface design, ease of use, comprehension, privacy, quality of on-line help and efficiency of screen organization. Optimal designs may be dependent on culture and education. Some of our research is designed to identify problems arising from the design of computer-based questionnaires.
Cognitive interviewing methodology identifies problems that are missed by traditional field tests. Field interviewers may not be sufficiently trained to identify questionnaire problems, and such tests are often conducted too late to allow for substantial revisions to be made. Nevertheless, field tests are a vital complement to cognitive interviews because they provide a better understanding of the magnitude of a problem. As time and resources allow, the behaviors of both interviewers and survey respondents in such interviews are observed and manually or audio recorded to allow for systematic analysis. These activities were used successfully to develop the questionnaires used in previous NHIS Supplements. The CCQDER therefore plans to apply these techniques in development of the NHIS revised Periodic and Topical Modules (formerly referred to as Supplements) and of modules from other surveys.
Generally, field tests/pilot interviews for face-to-face surveys are conducted in the respondent's household, and pilot interviews for telephone surveys are conducted over the telephone. Professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) conduct these interviews. A subset of these interviews may be observed by a survey professional (NCHS staff member or member of a Federal agency sponsoring the questionnaire). In cases involving observation, as the interviewer conducts the pilot interview, the observer compiles notes regarding respondent misunderstandings or difficulty answering, or questions that interviewers have difficulty administering, which help to identify potential question revisions. In addition, NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data. Subject matter staff are debriefed on these findings and the results of the field test/pilot interviews will be used to modify the questionnaire for follow-up field tests/pilot interviewing prior to the actual survey being conducted.
The practice of conducting field tests/pilot interviews allows testing of types of individuals who do not ordinarily volunteer for cognitive interviews in the laboratory; it also provides information collected under realistic field conditions, and collected early enough to be useful for questionnaire design decisions. This testing will be referred to in this document as field tests/pilot interviewing.
In addition to the applied questionnaire development activities described above, CCQDER designs, conducts, and manages research studies on the cognitive and interpretive aspects of survey methodology more generally. Such research could take the form of experiments embedded within fielded surveys (generally referred to as “split-ballot” experiments), experiments conducted in the laboratory, or exploratory studies employing individual interviews or focus groups. The purpose of the research is to enhance our understanding of the question response process, to develop better standards for questionnaire design, or to improve data collection procedures. Ultimately these studies produce generalizable knowledge that improves the quality of data collection instruments more generally.
Items of Information to be Collected
This clearance request is for continuing the five types of activities that NCHS and the CCQDER carries out: 1) Survey questionnaire development and testing for CDC, other federal agencies, or other academic or professional institutions based on cognitive interviewing methodology; 2) Research on the cognitive and interpretive aspects of survey methodology; 3) Research on computer-user interface design for computer-assisted instruments e.g. Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self-Interviewing (CASI) instruments including Web-based surveys and Audio Computer-Assisted Self-Interview (ACASI), as well as usability testing; 4) Field tests/pilot interviews (in person, or via telephone or virtual) are conducted with respondents, using professional field interviewers; 5) Studies of the optimal design and presentation of statistical graphical and textual material. See below for detailed explanations of these activities. Specific topics are addressed in individual collection requests under the generic clearance.
Information in Identifiable Form
Information in identifiable form (IIF) is collected for linkage of various CCQDER forms (informed consent documentation, and respondent demographics) and audio and video recordings. All of these items have been routinely approved and collected in the past. The identifiable information includes:
Name
Mailing Address
Email Address
Phone Number
Employment Status
Audio Identifier (digital voice recording)
Photographic Identifier (digital video image)
Access to personal information is restricted to CCQDER staff and designated agents (who have signed a Designated Agent Agreement) who can only access the personal information for statistical purposes.
2. Purpose and Use of Information Collection
NCHS and the Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) conducts cognitive interviews, focus groups, field tests/pilot interviews, and experimental research in laboratory and field settings, both for applied questionnaire evaluation and more basic research on response errors in surveys.
The purpose and use of collecting this information fall into six categories:
Development and testing of specific survey questionnaires
Research on the cognitive and interpretive aspects of survey methodology
Research to examine the ways in which social structure (e.g., gender, race, education, income) and the intersectionality of those constructs at the individual level impact the question response process and data quality
Research on human-computer interfaces/usability
Field tests/pilot interviewing
Studies of the optimal design and presentation of statistical graphical and textual material.
Development and cognitive testing of specific survey questionnaires:
The purpose of cognitive testing is not to obtain survey data, but rather to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the respondent, or otherwise contribute to the non-sampling errors of the survey.
Data collection procedures for cognitive interviewing are different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves: how respondents interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methods. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys. Some examples of previous and potential collections under this clearance include:
National Health Interview Survey (NHIS) (OMB No. 0920-0214, Exp. Date 12/31/2023): The NHIS collects annual data on health status and limitations, use of health care, AIDS testing, family resources, health insurance, access to care, injury, health behaviors and functioning. Personal interviews are conducted in approximately 43,000 households including about 106,000 persons. The CCQDER has routinely conducted cognitive testing of various modules under 10-day packages since 1999, including mental health, alternative health, disability, insurance, strengths and difficulties services, cancer screening questions, complementary and alternative medicine, oral health, children’s mental health, voice, swallowing, speech and language, sexual identity, health insurance, cancer control, occupational health, diabetes primary prevention, and cognitive disability. In 2012, the CCQDER conducted a 50 case, a 575 case, and a 5600 case field test in conjunction with the NHIS. The 50 case test bridged early cognitive testing work conducted by the CCQDER on the sexual identity questions with NHIS field interviews and focused on the programming of the ACASI portion of the questionnaire and the transition from orally administered to self-administered questions. The 575 case test employed field procedures consistent with the production NHIS, providing a more realistic reflection of the field effort required to obtain completed ACASI interviews and respondent acceptability of ACASI. The 5,600-case field test entailed a full scale dress rehearsal employing a nationally-representative sample with a goal of 5,600 completed interviews. As with the previous test, impacts of the new content/ACASI module on response rates, break-offs, and key NHIS indicators (critical for monitoring trends using existing NHIS measures) were monitored, and a systematic assessment of response error in the sexual identity measure were performed. Prevalence estimates for the sexual identity questions were compared by mode of administration. In addition, direct and indirect indicators of data quality, such as item nonresponse and response times, were compared between the CAPI and ACASI administrations.
It is anticipated that the CCQDER will conduct cognitive testing for numerous modules as well as additional field tests in 2021-2023.
Other parts of the Department of Health and Human Services (DHHS), such as the Assistant Secretary for Planning and Evaluation (ASPE) occasionally request that NCHS include new policy relevant questions on the NHIS. Examples included citizenship status, health insurance coverage, and the receipt of government services by low-income individuals and families. The CCQDER assisted in the development and pretesting of the survey questions. Assignments like this may occur during the 2018-2021 period and will be handled in a similar way. Further, in cases in which it may be difficult to identify and recruit the appropriate respondents (for example, persons who are undocumented aliens), contractors who have expertise in the use of cognitive techniques with difficult-to-locate populations will be enlisted to conduct the research with oversight by CCQDER staff.
Pregnancy Risk Assessment Monitoring System (PRAMS) (OMB No. 0920-0654): PRAMS is a surveillance project of the Centers for Disease Control and Prevention (CDC) and state health departments. PRAMS collects state-specific, population-based data on maternal attitudes and experiences prior to, during, and immediately following pregnancy.
The CCQDER has conducted cognitive testing on PRAMS questionnaires under 10-day packages in 1999, 2001, 2003, 2007, 2014 and 2016. We anticipate testing new questions, as well as questions that will be proposed as expansions and refinements to those already found in PRAMS questionnaires during 2018-2021.
National Health and Nutrition Examination Survey (NHANES) (OMB No. 0920-0950, Exp. Date 4/30/2023): NHANES collects annual data about the health and diet of people in the United States. The survey consists of two parts: an in-home interview and a health examination. The in-home interview asks questions about health status, disease history, and diet. The health examination consists of tests based on age and gender and is performed in a Mobile Examination Center.
The CCQDER has conducted cognitive testing of various modules under 10-day packages since 1999, including cognitive testing of a brochure designed to be used by the field interviewers to convert survey refusals, various modules for the “in-home interview,” including sexual orientation, physical activity and pain, positive prostate specific antigen (PSA), hypertension and pre-hypertension, audio-CASI sensitive questions, creatine & life style questions, second-hand smoke , health care utilization, smoking, alcohol intake and second hand e-cigarettes. It is anticipated that the CCQDER will conduct testing for numerous modules during 2021-2023.
National Survey of Family Growth (NSFG) (OMB No. 0920-0314, Exp. Date 06/30/2021): The National Survey of Family Growth (NSFG) is a multipurpose survey based on personal interviews with a national sample of men and women 15-49 years of age in the civilian non-institutionalized population of the United States. Its main purpose is to provide reliable national data on marriage, divorce, contraception, infertility, and the health of adults and infants in the United States.
The CCQDER has conducted cognitive testing of various NSFG modules under 10-day packages since 1999. It is anticipated that the CCQDER will conduct testing for numerous modules during 2021-2023.
Division of Health Care Statistics Surveys (DHCS) (various clearances):
The NCHS Division of Health Care Statistics includes surveys that are designed to answer key questions of interest to health care policy makers, public health professionals, and researchers. These can include the factors that influence the use of health care resources, the quality of health care, including safety, and disparities in health care services provided to population subgroups in the United States.
The CCQDER has conducted cognitive testing of the DHCS surveys and modules under 10-day packages including the National Ambulatory Medical Care Survey (NAMCS) and National Hospital Ambulatory Medical Care Survey (NHAMCS) Patient Record Evaluation Study; 2011 Physician Workflow Electronic Health Records (EMR) Supplement; 2012 Asthma Management Supplement; the National Survey of Long-Term Care Providers; 2015 NAMCS Feasibility Study, 2015 National Electronic Health Records Survey (NEHRS), and the 2016 NAMCS Culturally and Linguistically Appropriate Services (CLAS) Supplement. It is anticipated that the CCQDER will conduct testing for numerous modules during 2021-2023.
Research on Perceptions of Quality of Life: CCQDER staff will examine survey respondents’ perceptions of their self-assessed quality of life, and the basis for their responses to questions which purport to measure quality of life, especially from a health perspective. Such questions are increasingly important to both NCHS and CDC surveys as quality of life, rather then, simply length becomes a key measure. Questions from the Behavior Risk Factor Surveillance System (BRFSS) (OMB No. 0920-1061, Exp. Date 03/31/2022), Quality of Life Module will be subject to ongoing evaluation by CCQDER staff. In particular, cognitive testing will be conducted to determine whether modifications to question wording, response category ordering, and question re-ordering are likely to fundamentally affect the patterns of responses obtained. We also anticipate that several experts in the field of survey methodology and health assessment may be enlisted, under contracts, to assist in this research by, for example, conducting independent cognitive research, and comparing those results with those obtained in the NCHS Laboratory.
Other questionnaire testing and development: In addition to the specific questionnaire testing and development activities listed above, we anticipate that CCQDER staff will be asked over the next three years to test questionnaires developed by NCHS, other components of CDC, other Federal agencies, and possibly academic and professional institutions that collect data relevant to public health. It is appropriate that the CCQDER perform these activities, as it is currently the only Federal facility performing cognitive interviewing in order to develop DHHS survey questionnaires. However, because the requests may arrive with little advance notice, we cannot presently specify the nature of these questionnaires. Such a general plan was approved in the previous clearance, and the CCQDER was thus able to conduct quick response testing of questionnaires.
The information collections for questionnaire development activities above will usually be conducted in CCQDER facilities using cognitive interviewing procedures described in 11. In situations that prevent an in-person interviewing, such the 2019 COVID-19 pandemic these activities will be conducted virtually through video conferencing software. If we are unable to obtain adequate numbers of individuals from particular population subgroups (e.g., elderly, or those who have specific health problems), we will attempt to make arrangements with organizations such as centers for the elderly, or service organizations for persons with specific health conditions, to interview respondents at outside locations.
Traditionally, cognitive interviews have been conducted in-person. However, due to the current COVID-19 pandemic and travel restricts our cognitive interviews have been conducted virtually. For some projects we will attempt to match survey mode i.e., telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), web-based, or video-over-internet conferencing software, such as Skype, ZOOM, GoToMeeting, Lync, or WebEx. For a telephone interview or video-over-internet interview, we will conduct the interview in our laboratory, but calling/contacting the respondent from another laboratory room with face-to-face debriefing following. In the future, virtual interviews conducted over video conferencing software might become more prevalent as it has become suitable alternative during times of budget/travel restrictions and future pandemics.
Sometimes the NCHS CCQDER may collaborate with other agencies through interagency agreements. Consent forms will be modified to reflect their participation.
Research on the Cognitive and Interpretive Aspects of Survey Methodology
The second major purpose of the CCQDER’s data collection is to conduct research on the cognitive and interpretive aspects of survey methodology. Some examples of this methodological research include:
Research on appropriateness of response scales: An important determinant of survey data quality is that questions include appropriate response scales. In particular, response scales must have clear meanings to respondents, and must allow them to adequately express their experiences. An emerging body of research suggests that seemingly trivial variations in response scales (e.g., using a scale from 1 to 10 as opposed to a scale from
-5 to +5) can significantly affect response distributions. Preliminary research has also been conducted on the meanings of vague quantifiers (such as often, sometimes, and rarely) and the benefits of certain scales over others (e.g., seven-point scales over feeling thermometers). CCQDER staff will be engaged in additional research along these lines, possibly including cognitive laboratory testing of alternative response scales, as well as split-ballot experimentation.
Research on cognitive and interpretive aspects of nonresponse: Nonresponse creates numerous analytic difficulties on major surveys. Minimizing this problem requires a greater understanding of the cognitive processes that lead respondents to decide not to answer surveys or particular survey questions. CCQDER staff plan to conduct cognitive interviews using a variety of types of survey questions (behavioral and attitudinal) in order to explore these decision processes further. Survey nonresponse will be explored through an examination of reasons that nonresponders provide for their unwillingness or inability to complete surveys. It is also possible that data will be collected through experimental questionnaires administered outside of the laboratory that explore the effect of various design decisions on item nonresponse. Contracts may be used for some components of this data collection and analysis.
Respondent Perceptions of Confidentiality and Survey Participation: To encourage participation, NCHS surveys such as the NHIS and NHANES depend on advance letters, promising confidentiality and explaining uses of the data collected. However, it is not known how well these statements are generally understood, and believed, by survey respondents. Therefore, CCQDER staff proposes to conduct cognitive interviews of laboratory respondents in order to examine their comprehension of such statements. The results will be used to propose modifications to procedures used to communicate key issues related to informed consent, and to explain the need and purpose for survey data in a way intended to increase survey participation.
General Methodological Research: CCQDER staff constantly evaluate and refine the cognitive interviewing methods used at NCHS, especially in order to respond to changes such as the wide-spread introductions of CAPI (Computer Assisted Personal Interviewing) and Audio Computer-Assisted Self-Interview (ACASI) as a data collection tools. In addition, CCQDER staff may investigate the feasibility of using new technologies, such as video-over-internet conferencing software, such as Skype, ZOOM, GoToMeeting, Lync, or WebEx to conduct cognitive interviews, evaluate the quality of data obtained, and determine whether the mode of the interview impacts the interpretation of the survey questions. Further, CCQDER staff regularly conduct applied research on questionnaire design issues, such as the optimal wording for measures of complex concepts related to health status, utilization, and behavior. As CCQDER, continues to innovate and keep up with industry standards staff may also investigate recruitment and respondent participation at varying levels of incentive irrespective of interview mode, in an effort to establish empirical evidence regarding incentives and coercion.
In 2021-2024 NCHS/CCQDER staff plan to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, NCHS/ CCQDER will work collaboratively with survey researchers from universities and other Federal agencies to define and examine several research areas, including, but not limited to: 1) differences between face-to-face, telephone, and virtual/video-over internet cognitive interviewing, 2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing, 3) reactions of both survey respondents and survey interviewers to the use of Computer Assisted Personal Interviewing (CAPI), Audio Computer-Assisted Self-Interview (ACASI), video-over internet, 4) social, cultural and linguistic factors in the question response process, and 5) recruitment and respondent participation at varying levels of incentive in an effort to establish empirical evidence regarding their impact. Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, questionnaires that are of current interest (such as RANDS and NIOSH) may be evaluated using several of the techniques described above. Or, different versions of a survey question will be developed, and the variants then administered to separate groups of respondents in order to study the cognitive processes that account for the differences in responses obtained across different versions.
These studies will be conducted either by CCQDER staff, DHHS staff, or NCHS contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.
Research to examine the ways in which social structure (e.g., gender, race, education, income) and the intersectionality of those constructs at the individual level impact the question response process and data quality.
Central to the study of cognitive and interpretive processes, CCQDER studies set out to understand the ways in which social context impacts the question response process, thereby impacting comparability across relevant respondent groups. Specifically, CCQDER follows the socio-cultural approach as laid out in Miller and Willis (2016) and Miller et al. (2014)1.
This approach recognizes that the four-stage model of question response (i.e., comprehension, retrieval, judgment and response) developed within the field of psychology is an individual-centric understanding and does not fully account for the fact that these cognitions are informed by experience and meaning derived from social context. Therefore, to fully understand question performance, it is necessary to study how underlying social structures (and the intersectionality of those structures) inform meaning and shape the question response process. As such, the central research question within the socio-cultural approach is one of comparability: Do questions mean the same to all socio-cultural and linguistic groups represented in a survey? Are data elicited from questions capturing the same phenomena across all groups of respondents? Is the quality of responses similar across all respondent groups?
The information collections under this generic package will use a variety of methodologies consistent with the socio-cultural approach to question evaluation.
Cognitive Interviews, Focus Groups, and Usability Tests: Under this generic clearance qualitative studies—including cognitive interviewing/usability testing and focus group studies—are set within the socio-cultural approach and are specifically designed to address comparability. The design pertains to each stage of the research process, including data collection, analysis and documentation, and is fully detailed in Miller et al.’s book, Cognitive Interviewing Methodology: An Interpretive Approach for Survey Question Evaluation. For data collection, a purposive sample design, based on the constant comparative method, is implemented, and the principle of reflexivity is invoked for both interviewing procedures and analysis. Specifically, for analysis, a 5-stage process is utilized in which interpretive patterns are first identified across respondents and ultimately compared across relevant respondent groups, with a particular focus on race/ethnicity, education, income and gender. Final conclusions are publicly documented and reveal the phenomena (including unintended constructs) captured by each survey question, variations across respondent groups, and explanations for those variations.
Field Tests and Multi-Mode Question Evaluations (Including NCHS’ Research and Development Survey Program)
CCQDER’s recent expansion into more regularly using mixed-method question evaluation techniques via its Research and Development Survey (or RANDS) program also offers an opportunity to explore comparability—as defined above—within a quantitative design. Typically, the sample for RANDS is derived from a recruited, commercial survey panel. Doing so allows NCHS to collect a large enough sample to have the statistical power to analyze intersectional groups’ survey outcomes. When analyzing question evaluation data (such as that from web probes and embedded experiments), CCQDER is able to not only produce estimates of these groups’ patterns of interpretations and analyze how they differ from other subgroups, but also evaluate the magnitudes of these differences. Doing so can help survey programs determine how differential measurement error may impact the final estimates derived from the items the evaluated items.
Additionally, given that these field tests will be typically derived from standing commercial survey panels, the possibility of oversampling intersectional groups exists if needed for a specific study. For instance, if cognitive interviewing (or other previous qualitative research) indicates that low-income respondents of a particular racial group understand survey items differently than members of other subgroups, CCQDER can oversample that intersectional group for the field test, potentially giving it the statistical power needed to conduct wording or other experiments that may lead to revisions that eliminate differential interpretation.
Research on human-computer interfaces/usability
The third major purpose of this data collection is to conduct research on computer-user interface designs for computer-assisted instruments, often referred to as “usability testing.” This research examines how survey questions, instructions, and supplemental information are presented on computer instruments (e.g., CAPI, Computer Assisted Self-Interviewing (CASI) instruments, ACASI, or web-based instruments) and investigates how their presentation affects the ability of users to effectively use and interact with these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions that are not to be read to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on. Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task.
Usability testing has many similarities to questionnaire-based cognitive research (described in Section 2.1), since it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat different, in that the typical user can be an interviewer (in the case of CAPI instruments) as well as a respondent (in the case of CASI/ACASI instruments). It also focuses more heavily on matters of formatting and presentation of information than traditional cognitive testing does.
Research Using Field Tests/Pilot Interviewing
The fourth major purpose of QDRL’s data collection is to apply unobtrusive field-based questionnaire evaluation techniques. The different questionnaires may be field tested/pilot-tested either individually or in groups, depending on developmental status of the instruments, the appropriateness of combining them, and their overall length. For most field tests, professional field interviews (such as Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) usually conduct approximately 200 pilot interviews (person/telephone). However, in 2012, a field test of almost 600 interviews, and a full-scale dress rehearsal field test of 5,600 interviews was approved. Similarly, we anticipate that larger sample sizes may be required for future specific projects. There are four possible components to the proposed form of testing: a) a limited number of interviews on a draft version of the questionnaires are conducted using household respondents, b) a subset of interviews may be observed by NCHS and other staff trained in observational techniques, c) NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data and d) the potential for inclusion of built-in experiments i.e., two different versions of particular questions, in order to determine which version functions better in the field environment and mode of administration where test cases will be randomly assigned to receive questions in two different modes, i.e., CAPI or ACASI to assess the impact of mode on prevalence estimates and data quality.
Studies of the optimal design and presentation of graphical and textual material
The final major purpose is related to the growth of the Internet for collecting data (including Web-based surveys), and in disseminating health information. NCHS is the Federal government’s principal health statistics agency, and is responsible for collecting and disseminating many reports and volumes of data annually. During the last few years, the techniques developed for determining whether respondents understand survey questions have been applied with great utility to studying whether statistical publications and Web releases are optimally clear. One project, for example, involved the development and testing of a brochure designed by staff of the National Health and Nutrition Examination Survey (NHANES) (OMB. No. 0920-0950, Exp. Date 11/30/2021) to convert refusals to acceptance. Another recent project involved testing and evaluation of different Health Surveillance Map formats (choropleth versus isopleth) to determine if they affect ability to extract information from the maps for the Division of Adult and Community Health/the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP), CDC. We anticipate that there will be more work of this type during 2021-2023.
The major activities outlined above have well-demonstrated practical utility. As a result of laboratory testing, questionnaires may produce substantially less response error than would occur in the absence of this testing. Thus, users of NCHS data with measures that have been evaluated, in both Federal agencies and in the general health research community, will be less likely to be misled by erroneous statistical results. This assertion is supported by twenty four years of experience in using these techniques, and has been supported by findings presented at many statistical and research related conferences such as Joint Statistical Meetings (JSM), American Association for Public Opinion Research (AAPOR), American Sociological Association, and published in scientific journals such as “Applied Cognitive Psychology”, “Journal of Official Statistics”, “Public Opinion Quarterly”, “Field Methods”, and “Quality and Quantity”. The practical utility of field tests/pilot interviewing has also been supported in findings reported at an annual meeting of the American Statistical Association. Further evaluation of the efficacy of this method will be ongoing.
3. Use of Improved Information Technology and Burden Reduction
Usually, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), or web-based.
In addition, CCQDER Staff use searches in Q-Bank to determine if a survey question has been cognitively tested. The regular use of Q-Bank reduces unnecessary testing as well as allows CCQDER to build upon existing knowledge learned from past testing projects. Additionally, each cognitive interview is digitally recorded and stored on an internal, searchable video database. Like Q-Bank, this technology allows CCQDER staff to build upon past projects and, at the same time, it improves the accountability of test findings.
4. Efforts to Identify Duplication and Use of Similar Information
The CCQDER at NCHS is the only government facility that currently conducts testing and development of NCHS or other CDC questionnaires. Similar facilities at the Bureau of the Census and the Bureau of Labor Statistics bear the responsibility for testing survey questionnaires associated with their own agencies. The demand for CCQDER activities exceeds available resources.
In order to identify duplication across federal agencies, CCQDER hosts a publicly accessible online searchable database, Q-Bank, that contains all CCQDER evaluation reports. CCQDER encourages all agencies to submit their evaluation reports so that it is possible to track the work done across agencies as well as to build in existing knowledge.
5. Impact on Small Businesses and Other Small Entities
In the past, representatives of small businesses have been interviewed as part of testing of establishment surveys, such as the National Employer Health Insurance Survey (NEHIS) (OMB No. 0920-0341, Exp. Date 12/31/1994). If such requests are made, these businesses will be approached in the same manner as the individuals we normally recruit; we will ask the organization to identify the appropriate staff members with whom to conduct the cognitive interviews.
6. Consequences of Collecting the Information Less Frequently
Individual projects usually involve one-time data collection activities. There are no legal obstacles to reducing the burden.
7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5
This request fully complies with the regulation 5 CFR 1320.5.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies
A Federal Register notice for this collection was published on May 26, 2021 (Vol. 86, No. 100, p. 28357). The text of the notice is contained in Attachment B. CDC received one public comment related to this notice (see Attachment B1). No changes to the information collection plan were made in response to the public comment.
Consultants outside of CDC:
The following individuals have been consulted within the past year on survey methodology and/or on a specific project:
Gordon Willis
Program Director
Tobacco Control Research Branch (TCRB)
Behavioral Research Program (BRP)
National Cancer Institute
9609 Medical Center Drive, Rm 3E616, MSC 9762
Bethesda, MD 20892-9762
Alisu Schua-Glusberg
Research Support Services
906 Ridge Avenue58
Evanston, IL 60602
(847) 864-5677
Jennifer Hunter
Statistical Research Division
U.S. Bureau of the Census
Washington D.C. 20233
Consultants within CDC:
The following individuals have been consulted within the past year on survey methodology and/or on a specific project:
David Woodwell
Division of Health Care Statistics
3311 Toledo Road
Hyattsville, MD 20782
(301) 458-4327
Jim Dahlhamer
Division of Health Interview Statistics
3311 Toledo Road
Hyattsville, MD 20782
(301) 458-4403Phone
9. Explanation of Any Payment or Gift to Respondents
For most testing projects, cognitive interview respondents receive incentives for several reasons:
· Typically, respondents are recruited for specific characteristics that are related to the subject matter of the survey (e.g., questions may be relevant only to people with certain health conditions). The more specific the subject matter, the more difficult it is to recruit eligible respondents. Incentives helps to attract a greater number of potential respondents.
· Cognitive interviews require an unusual level of mental effort, as respondents are asked to explain their mental processes as they hear the question, discuss its meaning and any ambiguities, and describe why they answered the questions the way they did.
· They are usually asked to travel to the laboratory testing site, which involves transportation and parking expenses. (Many respondents incur additional expenses due to leaving their jobs during business hours, making arrangements for child care, etc.).
For a standard cognitive interviewing project in which one-hour interviews are conducted at NCHS and eligibility requirements are of average complexity, respondents will be given $40.00. Higher incentives may be requested on a case-by-case basis for particularly difficult recruitments. For example, in a 2008 & 2009 study, the CCQDER was unable to find epidemiologists willing to be interviewed for less than $75, and in 2011, 2013, and 2016 the CCQDER was unable to find physicians willing to be interviewed for less than $100. On rare occasions, a lower incentive is proposed.
It is important to offer incentives sufficient to attract the full range of needed respondent types for cognitive interviewing projects. Inadequate respondent recruitment limits the effectiveness of the questionnaire evaluation. Requests and justification for incentives will be included in each individual collection submission.
In the future, CCQDER plans to use e-gift cards as an alternative to our current method of giving incentive. As use of e-gift cards would better accommodate the current remote environment and provide an additional avenue for incentives that are comparable to industry standards.
For activities that are meant to resemble the usual household interview (for example, field tests/pilot interviewing), respondents will not receive incentives.
10. Protection of the Privacy and Confidentiality of Information Provided by Respondents
The NCHS Privacy Act Coordinator has reviewed this request and has determined that the Privacy Act is applicable. The related System of Records Notice is 09-20-0164 Health and Demographic Surveys Conducted in Probability Samples of the U.S. Population.
A Privacy Impact Assessment was submitted on May 16, 2016. The CCQDER continues to collect, on a confidential basis, data needed in order to conduct CCQDER studies. The process of informing respondents of the procedures used to keep information confidential begins with the telephone screener and will carry through to the interviewer and all communications with potential respondents. Materials will include all elements of informed consent, including the purpose of the data collection, the voluntary nature of the study, audio or video recording of the interview, and the effect upon the respondent for terminating the interview at any time.
Confidentiality provided to respondents is assured by adherence to Section 308(d) of the Public Health Service Act (42 U.S.C. 242m) which states:
"No information, if an establishment or person supplying the information or described in it is identifiable, obtained in the course of activities undertaken or supported under section...306 (NCHS legislation),...may be used for any purpose other than the purpose for which it was supplied unless such establishment or person has consented (as determined under regulations of the Secretary) to its use for such other purpose and (1) in the case of information obtained in the course of health statistical or epidemiological activities under section...306, such information may not be published or released in other form if the particular establishment or person supplying the information or described in it is identifiable unless such establishment or person has consented (as determined under regulations of the Secretary) to its publication or release in other form,..."
In addition, legislation covering confidentiality is provided according to section 513 of the Confidential Information Protection and Statistical Efficiency Act or CIPSEA (PL 107-347) which states:
“Whoever, being an officer, employee, or agent of an agency acquiring information for exclusively statistical purposes, having taken and subscribed the oath of office, or having sworn to observe the limitations imposed by section 512, comes into possession of such information by reason of his or her being an officer, employee, or agent and, knowing that the disclosure of the specific information is prohibited under the provisions of this title, willfully discloses the information in any manner to a person or agency not entitled to receive it, shall be guilty of a class E felony and imprisoned for not more than 5 years, or fined not more than $250,000, or both.”
The CIPSEA legislation authorizes the designation of agents (“designated agents” or “agents”) to perform statistical activities on behalf of an agency. These agents function under the supervision of the agency’s employees and are subject to the same provisions of law with regard to confidentiality as an agency’s employees. A Designated Agent Agreement between the agency and the designated agents (e.g. contractors) must be executed before the agents can acquire information for the agency for exclusively statistical purposes under a pledge of confidentiality. This requirement is outlined in an OMB Notice, published in the Federal Register on June 15, 2007, entitled “Implementation Guidance for Title V of the E-Government Act, Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA).”
A Designated Agent Agreement between NCHS and any CCQDER contractor will be executed if any contractors are hired to acquire information for the NCHS for exclusively statistical purposes under a pledge of confidentiality (i.e. complete any of the five types of activities described in this generic clearance request). Additionally, the agents (contractors) will be required to complete NCHS Confidentiality Training (https://www.cdc.gov/nchs/training/confidentiality/training/), submit a certificate of completion, and sign a pledge to maintain confidentiality (Nondisclosure Affidavit; see Attachment C) prior to completing CCQDER work. If the CCQDER contractor hires subcontractors to complete CCQDER work, the subcontractors must adhere to the same confidentiality and security requirements as CCQDER staff and contractors.
Data in identifiable form is collected for linkage of various CCQDER forms (informed consent documentation and respondent demographics) and audio and video recordings. The CCQDER also uses some identifiable data (name, phone number, email address) to contact previous respondents for CCQDER studies. The ability to match respondents to other data (informed consent documents, respondent demographics, and audio/video recordings) greatly expands the usefulness of the data at a very low cost.
As outlined in the informed consent form, access to personal information is restricted to CCQDER staff who can only access the personal information for statistical purposes. Additionally, designated agents such as CCQDER contractors or subcontractors may access the personal information for statistical purposes only after signing a Designated Agent Agreement with NCHS. CCQDER staff, designated agents, and staff from collaborating agencies must complete annual NCHS confidentiality training (https://www.cdc.gov/nchs/training/confidentiality/training/), submit a certificate of completion, and sign the NCHS affidavit of nondisclosure (see Attachment C) prior to being granted access to any personal information.
The collection of information in identifiable form requires strong measures to ensure that private information is not disclosed in a breach of confidentiality. Storage of confidential data is protected through procedures such as an internal QDRL LAN, passwords and restricted access.
Confidentiality of responses and safeguarding of data at NCHS
The CCQDER has a routine set of administrative, technical, and physical measures to safeguard confidentiality, including the following:
Storage of confidential data (informed consent form, respondent database, video and audio recordings) on the QDRL LAN are protected through procedures such as an internal QDRL LAN, passwords, and carefully restricted access;
The QDRL LAN is not located on the NCHS LAN, the QDRL LAN is inaccessible to others (not CCQDER staff) inside or outside NCHS;
All CCQDER personnel (including CCQDER contractors/designated agents) who have access to confidential data (informed consent form, respondent database, video and audio recordings) complete NCHS Confidentiality Training (https://www.cdc.gov/nchs/training/confidentiality/training/), submit a certificate of completion, and sign a pledge to maintain confidentiality (Nondisclosure Affidavit; see Attachment C), and are given instruction by the CCQDER Laboratory Manager on the requirement to protect confidentiality. Contracted personnel send hardcopies of the NCHS Confidentiality Training certificates and original signed hardcopies of the Nondisclosure Affidavits to Lauren Creamer, CCQDER Behavioral Scientist/Contracting Officer Representative (COR);
Only such authorized CCQDER personnel are allowed access to confidential data (informed consent form, respondent database, video and audio recordings) and only when their work requires it. CCQDER Personnel holding proper passwords may access the QDRL LAN through their CCQDER Computer Desk Top which is hardwired to the QDRL LAN;
Data (informed consent form, audio recordings) from cognitive interviews and focus groups that are conducted off-site are stored in a secured travel case to ensure that there is no loss in transit until returned to NCHS, at which point the data is stored in secure conditions (CCQDER Control Room or locked staff office in a locked drawer) until the recordings can be manually ingested and consent documents scanned into the secure QDRL LAN;
Restricted signage (see below) is placed on the external doors of the CCQDER with point of contact information and phone numbers of whom to contact during and after business hours.
QDRL Lab Access Protocol
This Lab is a restricted access secure facility. Access is by CCQDER staff or by CCQDER staff escort only.
Should access to the Lab be needed by anyone other than CCQDER staff members for any reason including emergency please adhere to the following:
During normal working hours contact one of the following CCQDER staff who will provide escorted access to the Lab.
Sean Murphy x4391 Mobile: 202-503-0321
Kristen Miller x4625 Mobile: 301-275-8182
Amanda Titus x4579 Mobile: 240-543-9171
During off hours, weekends, and holidays contact one of the following CCQDER staff:
Sean Murphy Mobile 202-503-0321
Lee Burch Mobile 301-233-0311
Kristen Miller Mobile 301-605-5350
All respondents receive a copy of Attachment E, Informed Consent Form, which describes the procedures by which confidentiality of data identifying individuals is maintained.
Q-Video Access Protocol
Only NCHS onsite CCQDER staff and contractors with PIV cards and proper passwords have access to the digitized video/audio application that captures, stores, and indexes the video and audio of a cognitive interview at the questionnaire level in a digitized database. Q-Video is housed on the QDRL LAN which is located in the secured QDRL Control Room within CCQDER’s secured space (authorized keycard access only). Q-Video is isolated on the QDRL LAN and it has no outside connectivity, including no connectivity to the NCHS LAN, CDC, or the internet. It is inaccessible to anyone who is not a CCQDER staff member or CCQDER contractor.
NOTE: While CCQDER staff and contractors have access to Q-Video they rarely access Q-Video directly. The most common method of viewing a video recording is by logging on to Q-Notes Plus which contains embedded video of the interview for which the CCQDER researcher is entering notes. The video is searchable to the specific question desired which is helpful in ensuring that the notes are accurate and correctly represent the interview as well as in analysis and report writing.
Q-Notes Access Protocol
Q-Notes is the external version of our notes software. It is accessible by CCQDER staff, contractors, sub-contractors (i.e. Designated Agents) for remote work on CCQDER projects, or anyone wishing to utilize Q-Notes for their own projects. Access for CCQDER staff and contractors is based upon a Principal Investigator and supporting research staff being assigned to a Project by the CCQDER Director. The CCQDER Principal Investigator (or CCQDER contractor for the Principal Investigator) sets themselves up as the “Project Manager” in Q-Notes for the specific project to which they were assigned. Then they can add staff to the project and assign staff appropriate access to that project. Once assigned to the project CCQDER staff, contractors, and subcontractors are required to enter passwords to log on to the project. In addition, outside organizations or individuals can register online to get their own workspace where they can create a project that is only accessible to them. Once they set themselves up as a “Project Manager” they can then assign staff to their project. The set up process, staff assignment, password access, and limitation of access to the project to which they are assigned is the same as it is for CCQDER. No one has access to a project to which they have not been assigned. They are only aware of their own project in Q-Notes. They are not aware of other projects existing.
Q-Notes Plus Access Protocol
Only NCHS onsite CCQDER staff and contractors (i.e. Designated Agents) with PIV cards and proper passwords have access to Q-Notes Plus (the internal interviewer notes application plus video/audio recordings). The process for setting up a project and granting staff access rights to a project is the same as it is for Q-Notes. Q-Notes Plus is housed on the QDRL LAN which is located in the secured QDRL Control Room within CCQDER’s secured space (authorized keycard access only). Because the QDRL LAN is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS. Access is further restricted at the cognitive interviewing project level. Only CCQDER staff and contractors assigned to a specific project have access to that projects’ interviewer notes and video/audio recordings.
Q-Bank Access Protocol
Q-Bank is accessible to the general public. There is no restricted access. Q-Bank is a database of evaluated questions from Federal surveys and the scientific report that evaluated the survey questions.
Records Retention Schedule for Cognitive Interviews
The cognitive interview’s retention status pertains to 1) the permitted retention time for its recording (e.g., two or five years), 2) the required media format for its storage (i.e. audio and/or video), and 3) the persons permitted access to the recording (e.g. special consent). An interview’s Retention Status is determined by several project and interview-level factors. Thus, each interview has its own individual retention status. Only interviews in which the respondent has consented to their interview being used for future research will be retained after the completion of the project. Future research consists only of work directly related to the survey questions discussed in the interview, but not necessarily tied to the current specific project. In most cases, retention is used to verify the accuracy of findings stated in final reports and allows for re-investigation of such findings. For those interviews being maintained for future use, the data retention period for storing the interview recording will begin after the conclusion of each project.
The factors determining retention status include the type of consent agreed upon by the respondent (e.g., special consent) and whether it is considered to be a restricted or unrestricted interview. Interviews become restricted depending on the type of respondent (e.g. < 18 years old) as well as the interview topic (e.g. illegal behaviors clearly defined by law and are punishable if disclosed). Restricted interviews require enhanced protections for data storage and retention. Interviews that are given the restricted designation are stripped of video (upon submission of the final report) and maintained only in audio format. The audience for restricted interviews is limited and the interviews are reviewed every two years to determine whether the interview continues to have qualitative value for use in federal question evaluation research projects or activities.
Unrestricted interviews contains questions about behaviors that are not illegal and in some projects, could be deemed as embarrassing or disconcerting. Unrestricted interviews can be maintained in audio and/or video format depending on respondent’s consent. The audience for unrestricted interviews is broader than restricted interview and may include sharing the interviews in a classroom setting if respondent’s consent was provided. Unrestricted interviews are reviewed every five years to determine whether the interview continues to have qualitative value for use in federal question evaluation research projects or activities.
If the restricted or unrestricted interview continues to be of value (defined as ongoing use by research staff, topic relevance, likely use for federal questions evaluation research), reassessment of the recording will occur again in either 2 years (for restricted interviews) or 5 years (for unrestricted interviews).
Informed Consent documents
Informed consent documents are stored by project in a separate drawer in a locked filing cabinet in the locked office of the CCQDER Recruiter or CCQDER staff member working on the project until the informed consent documents can be electronically scanned and moved to the secure CCQDER Local Area Network (LAN).
CCQDER Respondent Database: A custom-designed CCQDER Respondent Database contains personal identifiable information and demographic information on respondents who have participated in past CCQDER studies such as name, phone number, email address, age, marital status, ethnicity, race, education, employment status, and household income. The CCQDER Respondent Database is used to produce periodic tabulations and reports on database characteristics and response rates. The CCQDER Respondent Database is also used to conduct computerized searches to locate computer records of past respondents having salient characteristics for use in future studies. The CCQDER Respondent Database is housed on the secure QDRL LAN. Only CCQDER Staff and designated agents such as CCQDER contracted staff holding proper passwords have access to CCQDER Respondent Database. The QDRL LAN is located in the secured QDRL Control Room within CCQDER’s secured space (authorized keycard access only). Because the QDRL LAN is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS. In the event that normal office operations are restricted, such as the COVID-19 pandemic, CCQDER’s Respondent Database is to be housed on the NCHS CIPSEA sever. Access to the server is restricted to four members of CCQDER’s Operations Team that have undergone training in handling and protection of personal identifiable information.
Audio and video recordings: The CCQDER Recruiter/CCQDER Staff label each audio and video recording by a unique respondent identifier number, date, time, and project title. No other identifying information is labeled on the recording. Audio and video recordings are housed on the secure QDRL LAN. Only CCQDER Staff and designated agents such as CCQDER contracted staff holding proper passwords have access to interview recordings. The QDRL LAN is located in the secured QDRL Control Room within CCQDER’s secured space (authorized keycard access only). Because the QDRL LAN is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS.
Safeguarding of video recordings viewed at locations other than NCHS: Depending on the project, sponsors and collaborators may be from CDC, and occasionally from other DHHS or outside Federal agencies. The Informed Consent Form is tailored to describe each project and will specify which agencies are collaborating in the research and which staff(s) may be [listening to/viewing] the recording. Any outside NCHS collaborator [listening to/viewing] the recording onsite at NCHS in secured the QDRL will be required to complete NCHS Confidentiality Training (https://www.cdc.gov/nchs/training/confidentiality/training/), submit a certificate of completion, and sign a pledge to maintain confidentiality (Nondisclosure Affidavit, Attachment C).
Reports and publications: No respondent names or other personal identifying information is included in any reports or publications of cognitive testing results.
Presentations: No respondent names or other non-photographic identifying information is included in any presentations of cognitive testing results. As outlined in the standard informed consent and the special consent for expanded use of video and audio recordings, CCQDER respondents have been informed that voice and face identifiers will remain on the recording and have granted permission for the audio or video recordings to be played either to individuals working closely on the project or at conferences, meetings, or in the classroom.
11. Institutional Review Board (IRB) and Justification for Sensitive Questions
Continuation of protocol #2016-16 Laboratory Based Questionnaire Design (CCQDER) was approved by the NCHS Research Ethics Review Board on October 25, 2019 (Attachment H-1). An extension of the protocol #2016-16 was granted on April 8, 2021 (Attachment H-2).
Informed Consent and Voluntary Nature
CCQDER respondents/interviews conducted at NCHS
Respondents are recruited through media advertisements, flyers, and word-of-mouth, and either call the CCQDER voice mail system or contact a person coordinating the recruitment. Data collection for this project is authorized under 42 U.S.C. 242k (Section 306 of the Public Health Service Act).
During the telephone screener (Attachment D), potential respondents are informed that answering the telephone screener questions to determine their eligibility for the study is completely voluntary. They are informed that we are required by law to use the information they provided in the telephone screener for statistical research only and to keep it confidential, and that the law prohibits us from giving anyone any information that may identify them without their consent. In addition, respondents who are determined to be eligible for the study are informed during the telephone screener that the information they provide during the cognitive interview is confidential.
Prior to the start of the cognitive interview, CCQDER respondents read and sign Attachment E, Informed Consent Form (written at an 8th grade reading level). There are five templates in the attachment to cover various consent situations. The consent form states that participation is voluntary, they are free to terminate the interview at any time, and if they do so, they will still receive the incentive. The consent form describes the purpose of the interview and recording, specifies that the recordings may be played for other staff working closely on that project, that voice and face identifiers will remain on the recording, and that they may be recognized by a staff member viewing or listening to the recording. Cognitive interviews deemed to be about illegal behaviors will not be video recorded, only audio recorded. Respondents are given a copy of the consent form, which contains contact information for the CCQDER Laboratory Manager, the NCHS Research Ethics Review Board (ERB), and the NCHS Confidentiality Officer.
At the close of the cognitive interview, a respondent may also be asked by the interviewer to sign Attachment F, the Special Consent for Expanded Use of Video and Audio Recordings Form. The purpose of this form is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used when the topic of the cognitive interview is considered to be an illegal behavior (self-report or proxy report) or in the case of interviews with minors (persons under the age of 18); recordings of interviews with minors will never be shown to others not included in the study staff. Respondents are given a copy of the form which contains information about how to contact the CCQDER Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer. If respondents grant Special Consent, recordings are kept for as long as there is a justifiable, scientific use for the recordings as determined by the NCHS Research Ethics Review Board.
CCQDER respondents/interviews conducted off-site2: Sometimes interviewers must travel to conduct cognitive interviews in these cases, a mutually agreeable location will be chosen. In all cases, extreme care is taken with audio and video recordings and any materials that contain personal identifiers such as the Informed Consent Form or the Special Consent for Expanded Use of Video and Audio Recordings Materials are then transported to the CCQDER, where standard procedures are followed.
CCQDER respondents/interviews conducted virtually: Respondents are recruited through media advertisements, flyers, and word-of-mouth, and either call the CCQDER voice mail system or contact a person coordinating the recruitment.
During the telephone screener (Attachment D, template 1), potential respondents are informed that answering the telephone screener questions to determine their eligibility for the study is completely voluntary. They are informed that we are required by law to use the information they provided in the telephone screener for statistical research only and to keep it confidential, and that the law prohibits us from giving anyone any information that may identify them without their consent. In addition, respondents who are determined to be eligible for the study are informed during the telephone screener that the information they provide during the cognitive interview is confidential.
Prior to the start of the cognitive interview, CCQDER respondents read and sign Attachment E, template 1, Informed Consent Form (written at an 8th grade reading level). The consent form states that participation is voluntary, they are free to terminate the interview at any time, and if they do so, they will still receive the incentive. The consent form describes the purpose of the interview and recording, specifies that the recordings may be played for other staff working closely on that project, that voice and face identifiers will remain on the recording, and that they may be recognized by a staff member viewing or listening to the recording. After the interview has concluded respondents will be given the thank-you letter signed by Director of NCHS (Attachment O), their incentive, and a copy of the informed consent document (Attachment
E, template 1), which contains contact information for the CCQDER Laboratory Manager, the NCHS Research Ethics Review Board (ERB), and the NCHS Confidentiality Officer.
At the close of the cognitive interview, a respondent may also be asked by the interviewer to sign Attachment F, the Special Consent for Expanded Use of Video and Audio Recordings Form. The purpose of this form is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); recordings of interviews with minors will never be played in public settings only viewed by the study staff. Respondents are given a copy of the form which contains information about how to contact the CCQDER Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer.
NCHS government issued encrypted laptops will be used to video and audio record the interviews. Due to the size of the video recordings, the internal drive of the encrypted laptop is not sufficient for storage of the recordings. Recordings will be saved to an NCHS government issued encrypted flash drive. The encrypted flash drive is FIPS 140-2 compliant and approved for use by OCISO.
CCQDER staff will also use the NCHS government issued encrypted laptops to input their interviewer notes into Q-Notes. Within 24 hours, a CCQDER staff member will review project specific interview notes and will delete any direct or indirect personal identifiable information (PII) if found.
Extreme care will be taken with all recordings and paperwork from the interviews conducted virtually. Recordings and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets. Once the video and audio recordings are transferred to the QDRL Network, the recordings will be deleted from encrypted flash drive. Once deleted, the files are no longer available for use.
Focus groups: In focus group settings, participants are together and obviously can hear each other’s comments, statements, and questions. Participants are told in their initial telephone screening interview that they will be participating in a discussion group with other volunteers. Before the group discussion begins, participants sign the Informed Consent Form (Attachment E Template 3) which is tailored to specify that they will be participating in a focus group. The Informed Consent also states that they will be asked to pick a name and put it on a name tag, and that they do not have to use their real name. It is the responsibility of the interviewer (usually referred to as a moderator when conducting a focus group) to instruct the group that the information discussed will be held confidential by NCHS staff and should be treated confidentially by all respondents. Participants are strongly urged to respect the privacy of the other respondents and not to discuss with others what was discussed by the group.
At the close of the focus group, participants may be asked by the moderator to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Respondents of Discussion Groups Form. The purpose of this form is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from a focus group. Use of this form is at the discretion of the moderator and is typically warranted if (1) the focus group demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used when the topic of the cognitive interview is considered to be an illegal behavior (self-report or proxy report) or in the case of focus groups with minors (persons under the age of 18); recordings of focus groups with minors will never be shown to others not included in the study staff. Participants are given a copy of the form which contains information about how to contact the CCQDER Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer. If participants grant Special Consent, recordings are kept for as long as there is a justifiable, scientific use for the recordings as determined by the NCHS Research Ethics Review Board. If any one respondent from the focus group does not grant special consent, the recording will not be used in this way.
Contractor conducted interviews
On the rare occasion when contractors (designated agents) are used to collect data as part of CCQDER projects, they are contractually bound by NCHS confidentiality provisions and must submit documentation concerning their safeguarding practices to NCHS prior to data collection. The documentation is reviewed by the NCHS Confidentiality Officer and the NCHS Information Systems Security Officer. This is standard NCHS practice and does not reflect a special CCQDER procedure. If recordings are to be shared with the contractor, a contract as well as a Designated Agent Agreement will be developed. The contractor employee will view the NCHS confidentiality training (https://www.cdc.gov/nchs/training/confidentiality/training/), submit a certificate of completion, and sign the NCHS non-disclosure statement (see Attachment C) before starting work on the project.
Field Tests/Pilot Interviews
For field test/pilot interviews of household and telephone respondents, standard operating procedures regarding informed consent and survey administration procedures specific to the survey being tested will be followed.
Most of the questionnaires currently proposed for study generally do not contain questions that are highly sensitive in nature. There are some exceptions, such as the National Survey of Family Growth (OMB No. 0920-0314, Exp. Date 06/30/2021), National Health Interview Survey (NHIS) (OMB No. 0920-0214, Exp. Date 12/31/2023) questions on income and HIV, and National Health and Nutrition Examination Survey (NHANES) OMB No. 0920-0950, Exp. Date 11/30/2021) questions on sexual behavior. Again, one purpose of pre-testing such questions is to determine means for fashioning these questions in such a way that sensitivity is minimized, and responses are valid.
12. Estimates of Annualized Burden hours and costs:
An average of 22,125 respondents (66,375 over the three-year period) participate in CCQDER activities in a given year and the average annual respondent burden is estimated to be 9,455 hours. For the full three years there will be an estimated 28,365 hours of burden. Annualized estimates of respondent burden for each of the questionnaire development studies, over the course of data collection, are provided below. For most questionnaire development studies it is anticipated that interviews will last one hour. For some questionnaire development studies, questionnaire administration is anticipated to frequently require less than an hour of a respondent’s time (for example, a fifteen-minute interview may be conducted), and in rare cases, the burden may be more than one hour. Because the hours per response in questionnaire development studies are expected to vary, we will select the final sample size for each project in such a way that the total burden hours do not exceed the estimate listed above. For focus groups, the usual amount of time is 90 minutes (1.5 hours) which includes instructions and ancillary paperwork.
For interviews in the laboratory, time required to travel to the lab is not covered, because distances and modes of transportation are unknown. No retrieval of information by respondents is anticipated; although it is possible that validation of data at some point may require respondents to check records, probably those kept at home. In that case, the study will be designed so that the response time includes record retrieval. All estimates are based on NCHS' past experience (1988 through 2020).
Estimated Annualized Burden Table
Types of Respondents |
Form Name |
Number of Respondents |
Number of Responses per Respondent |
Average hours per response (in hours) |
Total Burden Hours
|
Individuals or households |
Eligibility Screeners |
4,400 |
1 |
5/60 |
367 |
Individuals or households |
Developmental Questionnaires |
8,750 |
1 |
55/60 |
8,021 |
Individuals or households |
Respondent Data Collection Sheet |
8,750 |
1 |
5/60 |
729 |
Individuals or households |
Focus Group Documents |
225 |
1 |
1.5 |
|
Total |
|
22,125 |
|
|
9,455 |
Estimated Annualized Burden Costs to Respondents.
The average annual response burden cost for the CCQDER is estimated to be $255,945.85. The hourly wage estimate is based on the Bureau of Labor Statistics May 2020 National Occupational Employment and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm). There is no cost to respondents other than their time to participate.
Type of Respondent |
Form Name |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Individuals or households |
Eligibility Screeners |
367 |
$27.07 |
$9,934.69 |
Individuals or households |
Developmental Questionnaires |
8,021 |
$27.07 |
$217,128.47 |
Individuals or households |
Respondent Data Collection Sheet |
729 |
$27.07 |
$19,734.03 |
Individuals or households |
Focus Group Documents |
338 |
$27.07 |
$9,149.66 |
Total |
|
$255,946.85 |
||
13. Estimates of Other Total Annual Cost Burden to Respondents and Record keepers
There is no annual capital or maintenance costs to the respondent resulting from this collection of information.
14. Annualized Costs to the Federal Government
The cost to the government consists mainly of the salaries of the CCQDER staff that will (1) assist the questionnaire designers in the design of appropriate laboratory instruments, (2) recruit, schedule, and assist in interviewing volunteer respondents, and (3) assist in the analysis of the results and recommend changes in questionnaire wording.
Total annualized project costs are as follows:
NCHS costs for CCQDER staff to plan, conduct, and analyze the outcomes of the questionnaire development activities:
Staffing 17.0 FTEs $1,897,625.00
Incentives for CCQDER respondents 4500 @ $40
(Pilot Household and web panel tests do not include incentives) $180,000.00
CCQDER Contract Staff (including remuneration) $985,250.24
Contracts for assistance with
methodological research $30,000.00
Off-site travel: (see note below under
travel costs) $10,000.00
Materials for conducting household
interviews $500.00
Flyers $200.00
Advertisements $24,750.00
Hardware and software upgrades $50,000.00
Annual Total $3,178,325.24
3 Year Total (for generic submission) $9,534,975.72
Travel costs: Most data will be collected in NCHS office space. However, it will be more efficient in certain instances to hold interviews with individuals at other locations, which will involve some travel costs. Further, household interviews will require limited numbers of in-person interviews in respondent households. Household interviews will be done locally, in order to limit travel costs, unless there is a compelling reason to do otherwise (for example, if respondents critical to the study can be interviewed only at a distant location).
15. Explanation for Program Changes or Adjustments
This is a generic clearance. The current annualize burden is 7,783 hours, totaling 23,350 hours over three years. We are requesting 9,455 annualized hours (28,365 burden hours over three years); which is an increase of 1,672 hours per year for a total increase of 5,016 hours over three years. The program change is due to an anticipated increase in the number and size of projects being undertaken.
16. Plans for Tabulation and Publication and Project Time Schedule
This clearance request is for questionnaire development activities to be conducted prior to survey production and for developmental work that will guide future questionnaire design. The majority of laboratory investigations will be analyzed qualitatively. The survey designers and lab staff serve as interviewers and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. Final reports will be written that document how the question performed in the interviews, including question problems as well as the phenomena captured by the survey question. All reports will be placed on Q-Bank for public access. Reports are used to provide necessary information to guide designs for redesigning a question prior to fielding as well as to assist end users when analyzing the survey data. For field tests/pilot interviewing activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews in order to determine where additional problems occur. Because NCHS is using state-of-the-art questionnaire development techniques, methodological papers will be written which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the cognitive interview and observation techniques.
Each individually submitted information collection will include a project time schedule specific to that project.
17. Reason(s) Display of OMB Expiration Date is Inappropriate
The expiration date will be displayed.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
The certifications are included in this submission.
1 Miller, K. and Willis, G. (2016) Cognitive Models of Answering Processes. The SAGE Handbook of Survey Methodology. Thousand Oaks, CA: Sage.
Miller, K., Willson, S., Chepp, V., and Padilla, J. (2014) Cognitive Interviewing Methodology: An Interpretive Approach for Survey Question Evaluation. (New York: Wiley and Sons.
2Off-site interviews fall into two categories. First, it is not always feasible for individuals to travel to the CCQDER, or it may be more efficient for interviewers to travel to a particular site. Second, we occasionally conduct establishment studies where a visit to the business location is pertinent to data collection.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-08-17 |
© 2024 OMB.report | Privacy Policy