Supp Statement A 060815

Supporting Statement A for Request for Clearance:

QUESTIONNAIRE DESIGN RESEARCH LABORATORY

2015-2018

OMB No. 0920-0222

Contact Information:

Karen R. Whitaker, B.S.

Laboratory Manager, Questionnaire Design Research Laboratory

Office of Research and Methodology

National Center for Health Statistics/CDC

3311 Toledo Road, Room 3215

Hyattsville, MD 20782

301-458-4569

[email protected]

301-458-4031 (fax)

May 28, 2015

Table of Contents

A. Justification

A.1. Circumstance Making the Collection of Information Necessary 4

A.2. Purpose and Use of Information Collection 9

A.3. Use of Improved Information Technology and Burden Reduction 16

A.4. Efforts to Identify Duplication and Use of Similar Information 17

A.5. Impact on Small Businesses or Other Small Entities 17

A.6. Consequences of Collecting the Information Less Frequently 17

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 17

A.8. Comments in Response to the Federal Register Notice and

Efforts to Consult Outside the Agency 17

A.9. Explanation of Any Payment or Gift to Respondents 19

A.10. Assurance of Confidentiality Provided to Respondents 19

A.11. Justification for Sensitive Questions 25

A.12. Estimates of Annualized Burden Hours and Costs 25

A.13. Estimates of Other Total Annual Cost Burden to Respondents

or Record Keepers 26

A.14. Annualized Cost to the Government 26

A.15. Explanation for Program Changes or Adjustments 27

A.16. Plans for Tabulation and Publication and Project Time Schedule 28

A.17. Reason(s) Display of OMB Expiration Date in Inappropriate 28

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions 28

• Goal of the study: It is the goal of the QDRL to not only evaluate questions for optimal design, but also to provide documentation supporting the validity of NCHS survey data.

• Intended use of the resulting data: The purpose is to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the respondent, or otherwise contribute to the non-sampling errors of the survey.

• Methods to be used to collect: QDRL Staff use various techniques to design and evaluate interviewer administered, self-administered, telephone, Computer Assisted Personal Interviewing (CAPI) and Computer Assisted Self-Interviewing (CASI), Audio Computer-Assisted Self-Interview (ACASI), and web-based questionnaires.  These methods include cognitive interviewing, focus groups, usability testing, ethnography, and field tests/pilot interviews (personal/telephone/web). 

• Subpopulation to be studied:  For qualitative methods, respondents are not selected through a random process, but rather are selected for specific characteristics such as race or health status or some other attribute that is relevant to the type of questions being tested. 

• How data will be analyzed:  The goal of methods such as cognitive interviewing is to identify the presence of interpretive patterns as opposed to making estimations or causal statements.  Depending on the needs of the project, a variety of analysis methods may be used (such as statistical analysis of non-response), and will be explained in the individual information collection requests under this clearance.

Supporting Statement A

Questionnaire Design Research Laboratory

A three-year OMB clearance revision is requested for “NCHS Questionnaire Design Research Laboratory (OMB No. 0920-0222).” This generic clearance request encompasses general questionnaire development and pre-testing activities to be carried out in 2015-2018 in the Office of Research and Methodology, National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC). The activities are to be conducted by the staff of the Questionnaire Design Research Laboratory (QDRL) and involve the development of health-related survey questionnaires, using a methodology which has been employed effectively since 1985. This revision seeks only an increase in burden hours to allow the QDRL to conduct more studies.

A. JUSTIFICATION

1. Circumstances Making the Collection of Information Necessary

In 1983/1984, the Committee on National Statistics conducted a two-part seminar on the Cognitive Aspects of Survey Methodology (CASM) under a grant from the National Science Foundation (NSF). The respondents in the CASM seminar (CASM I) were survey researchers and cognitive psychologists from academic institutions and survey researchers from the National Center for Health Statistics (NCHS) and the Bureau of the Census. The seminar examined a number of cognitive-related methodological studies that might lead to improvements in the questionnaires and interviewing procedures employed in scientific surveys in general, and in the National Health Interview Survey (NHIS) as a test case.

Following this seminar, the NSF provided funding to NCHS to investigate how relevant knowledge and techniques in cognitive science could be applied to im­prove health surveys. The project, begun in 1984, was called Laboratory-Based Studies of the Cognitive Aspects of Survey Methodology (CASM), and used cognitive psychological methods to study the survey interviewing process. In its final report, NCHS concluded that it is feasible and efficient for a Feder­al statistical agency to conduct laboratory research on the cogni­tive aspects of survey questionnaires. Subsequently, NCHS applied the cognitive research techniques being tested under the grant to develop the 1987 NHIS supplement, a comprehensive set of questions on knowledge, attitudes, and practices regarding cancer risk factors. Cognitive research techniques (also known as cognitive interviewing) proved invaluable for identifying conceptual problems with draft questions. The NCHS project staff concluded from this experience that past questionnaire design procedures were often unable to identify questions that were failing to measure what was intended, but that interviews in the laboratory were effective for identifying these kinds of measurement errors. The Questionnaire Design Research Laboratory (QDRL) was created at NCHS to provide such testing for NCHS surveys on a regular basis, as well as to continue more general research on the survey response process, questionnaire design, and pretesting methodology.

In October 2009, NCHS held a Question Evaluation Methods Workshop to examine various question evaluation methods as well as to discuss the impact of question design and evaluation on survey data quality. Broad consensus determined that measurement error in the Federal statistical enterprise requires renewed consideration. Federal statistical agencies have a fundamental obligation to produce valid and reliable data, and more attention needs to be placed on question evaluation and documentation. Furthermore, it was established that validation of measures is a particularly complex, methodological problem that requires a mixed-method approach. While quantitative methods are essential for understanding the magnitude and prevalence of error, they remain dependent on the interpretive power of cognitive interviewing. Unlike any other question evaluation method, cognitive interviewing can portray the interpretive processes that ultimately produce survey data. As it is practiced as a qualitative methodology, cognitive interviewing reveals these processes as well as the type of information that is transported through statistics. Consequently, cognitive interviewing is an integral method for ensuring the validity of statistical data, and the documented findings from these studies represent tangible evidence of how the question performs.

It is the mission of the QDRL to not only evaluate questions for optimal design, but also to provide documentation supporting the validity of NCHS survey data. Specifically, the final product of each study conducted under the auspices of this clearance will be a clearly documented and publically available report. Such documentation also serves NCHS data users, allowing them to be critical users in their approach and application of the data. Consequently, all completed QDRL testing reports are located and made accessible on Q-Bank (http://cdc.gov/qbank), an interagency, online searchable database that houses question evaluation studies.

Data collection for this project is authorized under 42 USC 242k (Section 306 of the Public Health Service Act). A copy of the legislation is provided in Attach­ment A. CDC is requesting terms of clearance identical to previous submissions. CDC will submit individual collections under this generic three year clearance to OMB. It is requested that OMB continue to provide feedback on the individual collections within 10 working days of the submission. Standard remuneration of cognitive interviewing respondents will be capped at $40.00 for an hour interview, though higher remunerations may be requested with justification for difficult recruitments, such as medical doctors and practitioners, specialized populations, focus groups, etc.

Overview of the Data Collection System

QDRL Staff are methodological specialists who examine questionnaires from NCHS, CDC, other federal agencies, or other academic or professional institutions. Specific topics are addressed in individual collection requests under the generic clearance.

QDRL Staff use various techniques to design and evaluate interviewer administered, self-administered, telephone, Computer Assisted Personal Interviewing (CAPI) and Computer Assisted Self-Interviewing (CASI), Audio Computer-Assisted Self-Interview (ACASI), and web-based questionnaires. These methods include cognitive interviewing, focus groups, usability testing, ethnography, and field tests/pilot interviews (personal/telephone/web). Unless there are clear reasons for not making recordings, most evaluations of questionnaires will be video and audio recorded to allow researchers to review not only the transcript of the interaction, but also the behaviors and body language of the respondents. Interviews conducted offsite—outside of the physical Questionnaire Design Research Laboratory at NCHS— may be audio or video recorded. In general, these recordings will allow researchers to insure the quality of their interview notes.

For qualitative methods, respondents are not selected through a random process, but rather are selected for specific characteristics such as race or health status or some other attribute that is relevant to the type of questions being tested. Because the goal of methods such as cognitive interviewing is to identify the presence of interpretive patterns as opposed to making estimations or causal statements, a purposive sample rather than one randomly drawn, is utilized. However, for quantitative field tests, a wider range of sampling strategies will be employed—depending on the specific purpose of the test. Examples of these strategies might include Random Digit Dial (RDD) and web-based panels designed to match certain demographics.

To rigorously and systematically examine the cognitive interviewing, focus group, and usability testing data, online and internal applications have been developed. They include: Q-Video, Q-Notes, Q-Notes Plus and Q-Bank.

Q-Video is a digitized video/audio application that captures, stores, and indexes the video and audio of a cognitive interview at the questionnaire level in a digitized database for the purpose of searching individual questions and conducting analysis.

Q-Notes is a qualitative research application developed to assist in the management of and allow for more rapid, yet thorough systematic collection and analysis of cognitive interviewing studies. Q-Notes provides interviewers and analysts real-time access to interview data and allows interviews to be conducted in multiple geographical regions so that comparability can be examined for multi-national and multi-lingual surveys. Q-Notes is currently used by various academics and statistical agencies around the world both for their own projects and to collaborate with other agencies internationally.

Q-Notes Plus is an extension of Q-Notes, where the actual interview is embedded within the application so that the findings can truly be traced to the original source.

Q-Bank (a product of an interagency collaboration and hosted by NCHS) is a database consisting of evaluated questions from Federal surveys and links each question to the scientific report that evaluated the survey question. Questions are searchable by survey title, question topic (e.g. income, demographic, chronic health conditions), information type (e.g. objective characteristics, behavioral reports, attitudes), response category (e.g. yes/no, open-ended, quantity), and response error (e.g. problems with terms, recall problems). In addition, users can search for keywords within individual questions. Q-Bank is intended to help users of survey data interpret the survey questions on which the data are based and understand the potential errors that might be associated with these questions.

The most commonly used method under this general clearance will be cognitive interviewing. Cognitive interviews offer detailed depictions of meanings and processes used by respondents to answer questions—processes that ultimately produce the survey data. As such, the method offers insights that can transform the understanding of question validity and response error.

The interview structure tends to consist of respondents first answering a draft survey question and then providing textual information to reveal the processes involved in answering the test question. Specifically, cognitive interview respondents are asked to describe how and why they answered the question as they did. Through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered.

Occasionally, focus groups (or group interviews of 5-10 individuals) are used to discuss general concepts that survey questions will focus on. Individual interviews are generally preferable to focus groups for evaluating specific questions because respondents usually respond to surveys individually, and the group dynamic can have a strong influence on interpretations and responses. However, focus groups can sometimes help questionnaire designers to understand the circumstances of various groups of people, and this information can be used to craft questions that better match respondent experiences.

Additional issues arise in computer-assisted survey instruments. Issues include the human-interface design, ease of use, comprehension, privacy, quality of on-line help and efficiency of screen organization. Optimal designs may be dependent on culture and education. Some of our research is designed to identify problems arising from the design of computer-based questionnaires.

Cognitive interviewing methodology identifies problems that are missed by traditional field tests. Field interviewers may not be sufficiently trained to identify questionnaire problems, and such tests are often conducted too late to allow for substantial revisions to be made. Nevertheless, field tests are a vital complement to cognitive interviews because they provide a better understanding of the magnitude of a problem. As time and resources allow, the behaviors of both interviewers and survey respondents in such interviews are observed and manually or audio recorded to allow for systematic analysis. These activities were used successfully to develop the questionnaires used in previous NHIS Supplements. The QDRL therefore plans to apply these techniques in development of the NHIS revised Periodic and Topical Modules (formerly referred to as Supplements) and of modules from other surveys.

Generally, field tests/pilot interviews for face-to-face surveys are conducted in the respondent's household, and pilot interviews for telephone surveys are conducted over the telephone. Professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) conduct these interviews. A subset of these interviews may be observed by a survey professional (NCHS staff member or member of a Federal agency sponsoring the questionnaire). In cases involving observation, as the interviewer conducts the pilot interview, the observer compiles notes regarding respondent misunderstandings or difficulty answering, or questions that interviewers have difficulty administering, which help to identify potential question revisions. In addition, NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data. Subject matter staff are debriefed on these findings and the results of the field test/pilot interviews will be used to modify the questionnaire for follow-up field tests/pilot interviewing prior to the actual survey being conducted.

The practice of conducting field tests/pilot interviews allows testing of types of individuals who do not ordinarily volunteer for cognitive interviews in the laboratory; it also provides information collected under realistic field conditions, and collected early enough to be useful for questionnaire design decisions. This testing will be referred to in this document as field tests/pilot interviewing.

In addition to the applied questionnaire development activities described above, QDRL staff design and conduct research studies on the cognitive and interpretive aspects of survey methodology more generally. Such research could take the form of experiments embedded within fielded surveys (generally referred to as “split-ballot” experiments), experiments conducted in the laboratory, or exploratory studies employing individual interviews or focus groups. The purpose of the research is to enhance our understanding of the question response process, to develop better standards for questionnaire design, or to improve data collection procedures. Ultimately these studies produce generalizable knowledge that improves the quality of data collection instruments more generally.

Audio and video recordings will be kept indefinitely unless: 1) Special Consent for Expanded Use of Video and Audio Recordings has not been granted, or 2) Language in the consent form does not permit investigators to keep the recordings for future unspecified research.

Items of Information to be Collected

This clearance request is for continuing the five types of activities that NCHS and the QDRL carries out: 1) Survey questionnaire development and testing for CDC, other federal agencies, or other academic or professional institutions based on cognitive interviewing methodology; 2) Research on the cognitive and interpretive aspects of survey methodology; 3) Research on computer-user interface design for computer-assisted instruments e.g. Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self-Interviewing (CASI) instruments including Web-based surveys and Audio Computer-Assisted Self-Interview (ACASI), as well as usability testing; 4) Field tests/pilot interviews (in personal, or via telephone or the internet) are conducted with respondents, using professional field interviewers; 5) Studies of the optimal design and presentation of statistical graphical and textual material. See 2.1 - 2.5 for a detailed explanation of these activities. Specific topics are addressed in individual collection requests under the generic clearance.

Information in Identifiable Form

Information in identifiable form (IIF) is collected for linkage of various QDRL forms (informed consent documentation, and respondent demographics) and audio and video recordings. All of these items have been routinely approved and collected in the past. The identifiable information includes:

• Name

• Phone Number

• Employment Status

• Photographic Identifier (digital video image)

2. Purpose and Use of Information Collection

NCHS and the Questionnaire Design Research Laboratory (QDRL) conducts cognitive interviews, focus groups, field tests/pilot interviews, and experimental research in laboratory and field settings, both for applied questionnaire evaluation and more basic research on response errors in surveys.

The purpose and use of collecting this information fall into five categories:

• Development and testing of specific survey questionnaires

• Research on the cognitive and interpretive aspects of survey methodology

• Research on human-computer interfaces/usability

• Field tests/pilot interviewing

• Studies of the optimal design and presentation of statistical graphical and textual material.

Development and cognitive testing of specific survey questionnaires:

The purpose of cognitive testing is not to obtain survey data, but rather to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the respondent, or otherwise contribute to the non-sampling errors of the survey.

Data collection procedures for cognitive interviewing are different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves: how respondents interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methods. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys. Some examples of previous and potential collections under this clearance include:

National Health Interview Survey (NHIS) (OMB # 0920-0214): The NHIS collects annual data on health status and limitations, use of health care, AIDS testing, family resources, health insurance, access to care, injury, health behaviors and functioning. Personal interviews are conducted in approximately 43,000 households including about 106,000 persons. The QDRL has conducted cognitive testing of various modules under 10-day packages since 1999, including mental health, alternative health, disability, insurance, strengths and difficulties services, cancer screening questions, complementary and alternative medicine, oral health, children’s mental health, voice, swallowing, speech and language, sexual identity, health insurance, cancer control, and occupational health. In 2012, the QDRL conducted a 50 case, a 575 case, and a 5600 case field test in conjunction with the NHIS. The 50 case test bridged early cognitive testing work conducted by the QDRL on the sexual identity questions with NHIS field interviews and focused on the programming of the ACASI portion of the questionnaire and the transition from orally administered to self-administered questions. The 575 case test employed field procedures consistent with the production NHIS, providing a more realistic reflection of the field effort required to obtain completed ACASI interviews and respondent acceptability of ACASI. The 5,600-case field test entailed a full scale dress rehearsal employing a nationally-representative sample with a goal of 5,600 completed interviews. As with the previous test, impacts of the new content/ACASI module on response rates, break-offs, and key NHIS indicators (critical for monitoring trends using existing NHIS measures) were monitored, and a systematic assessment of response error in the sexual identity measure were performed. Prevalence estimates for the sexual identity questions were compared by mode of administration. In addition, direct and indirect indicators of data quality, such as item nonresponse and response times, were compared between the CAPI and ACASI administrations. 

It is anticipated that the QDRL will conduct cognitive testing for numerous modules as well as additional field tests in 2015-2018.

Other parts of the Department of Health and Human Services (DHHS), such as the Assistant Secretary for Planning and Evaluation (ASPE) occasionally request that NCHS include new policy relevant questions on the NHIS. Examples included citizenship status, health insurance coverage, and the receipt of government services by low-income individuals and families. The QDRL assisted in the development and pretesting of the survey questions. Assignments like this may occur during the 2015-2018 period and will be handled in a similar way. Further, in cases in which it may be difficult to identify and recruit the appropriate respondents (for example, persons who are undocumented aliens), contractors who have expertise in the use of cognitive techniques with difficult-to-locate populations will be enlisted to conduct the research with oversight by QDRL staff.

Pregnancy Risk Assessment Monitoring System (PRAMS) (OMB# 0920-0654: PRAMS is a surveillance project of the Centers for Disease Control and Prevention (CDC) and state health departments. PRAMS collects state-specific, population-based data on maternal attitudes and experiences prior to, during, and immediately following pregnancy.

The QDRL has conducted cognitive testing on PRAMS questionnaires under 10-day packages in 1999, 2001, 2003, 2007, and 2014. We anticipate testing new questions, as well as questions that will be proposed as expansions and refinements to those already found in PRAMS questionnaires during 2015-2018.

National Health and Nutrition Examination Survey (NHANES) (OMB # 0920-0950):

NHANES collects annual data about the health and diet of people in the United States. The survey consists of two parts: an in-home interview and a health examination. The in-home interview asks questions about health status, disease history, and diet. The health examination consists of tests based on age and gender and is performed in a Mobile Examination Center.

The QDRL has conducted cognitive testing of various modules under 10-day packages since 1999, including cognitive testing of a brochure designed to be used by the field interviewers to convert survey refusals, various modules for the “in-home interview,” including sexual orientation, physical activity and pain, positive prostate specific antigen (PSA), hypertension and pre-hypertension, audio-CASI sensitive questions, creatine & life style questions, and second-hand smoke questions. It is anticipated that the QDRL will conduct testing for numerous modules during 2015-2018.

National Survey of Family Growth (NSFG) (OMB # 0920-0314): The National Survey of Family Growth (NSFG) is a multipurpose survey based on personal interviews with a national sample of men and women 15-49 years of age in the civilian non-institutionalized population of the United States. Its main purpose is to provide reliable national data on marriage, divorce, contraception, infertility, and the health of adults and infants in the United States.

The QDRL has conducted cognitive testing of various NSFG modules under 10-day packages since 1999. It is anticipated that the QDRL will conduct testing for numerous modules during 2015-2018.

Division of Health Care Statistics Surveys (DHCS) (various clearances):

The NCHS Division of Health Care Statistics includes surveys that are designed to answer key questions of interest to health care policy makers, public health professionals, and researchers. These can include the factors that influence the use of health care resources, the quality of health care, including safety, and disparities in health care services provided to population subgroups in the United States.

The QDRL has conducted cognitive testing of the DHCS surveys and modules including the National Ambulatory Medical Care Survey (NAMCS) and National Hospital Ambulatory Medical Care Survey (NHAMCS) Patient Record Evaluation Study; 2011 Physician Workflow Electronic Health Records (EMR) Supplement; 2012 Asthma Management Supplement; and the National Survey of Long-Term Care Providers. It is anticipated that the QDRL will conduct testing for numerous modules during 2015-2018.

Research on Perceptions of Quality of Life: QDRL staff will examine survey respondents’ perceptions of their self-assessed quality of life, and the basis for their responses to questions which purport to measure quality of life, especially from a health perspective. Such questions are increasingly important to both NCHS and CDC surveys as quality of life, rather then, simply length becomes a key measure. Questions from the Behavior Risk Factor Surveillance System (BRFSS) (OMB# 0920-1061), Quality of Life Module will be subject to ongoing evaluation by QDRL staff. In particular, cognitive testing will be conducted to determine whether modifications to question wording, response category ordering, and question re-ordering are likely to fundamentally affect the patterns of responses obtained. We also anticipate that several experts in the field of survey methodology and health assessment may be enlisted, under contracts, to assist in this research by, for example, conducting independent cognitive research, and comparing those results with those obtained in the NCHS Laboratory.

Other questionnaire testing and development: In addition to the specific questionnaire testing and development activities listed above, we anticipate that QDRL staff will be asked over the next three years to test questionnaires developed by NCHS, other components of CDC, other Federal agencies, and possibly academic and professional institutions that collect data relevant to public health. It is appropriate that the QDRL perform these activities, as it is currently the only Federal facility performing cognitive interviewing in order to develop DHHS survey questionnaires However, because the requests may arrive with little advance notice, we cannot presently specify the nature of these questionnaires. Such a general plan was approved in the previous clearance (No. 0920-0222), and the QDRL was thus able to conduct quick response testing of questionnaires.

The information collections for questionnaire development activities (a) through (i) above will usually be conducted in QDRL facilities using cognitive interviewing procedures described in 2. If we are unable to obtain adequate numbers of individuals from particular population subgroups (e.g., elderly, or those who have specific health problems), we will attempt to make arrangements with organizations such as centers for the elderly, or service organizations for persons with specific health conditions, to interview respondents at outside locations.

Normally, cognitive interviews are conducted face-to-face. For some projects will be attempt to match survey mode i.e., telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), web-based, or video-over-internet conferencing software, such as Skype, GoToMeeting, Lync, or WebEx. For a telephone interview or video-over-internet interview, we will conduct the interview in our laboratory, but calling/contacting the respondent from another laboratory room with face-to-face debriefing following.

Sometimes the NCHS QDRL may collaborate with other agencies through interagency agreements. Consent forms will be modified to reflect their participation.

Research on the Cognitive and Interpretive Aspects of Survey Methodology

The second major purpose of the QDRL’s data collection is to conduct research on the cognitive and interpretive aspects of survey methodology. Some examples of this methodological research include:

Research on appropriateness of response scales: An important determinant of survey data quality is that questions include appropriate response scales. In particular, response scales must have clear meanings to respondents, and must allow them to adequately express their experiences. An emerging body of research suggests that seemingly trivial variations in response scales (e.g., using a scale from 1 to 10 as opposed to a scale from

-5 to +5) can significantly affect response distributions. Preliminary research has also been conducted on the meanings of vague quantifiers (such as often, sometimes, and rarely) and the benefits of certain scales over others (e.g., seven-point scales over feeling thermometers). QDRL staff will be engaged in additional research along these lines, possibly including cognitive laboratory testing of alternative response scales, as well as split-ballot experimentation.

Research on cognitive and interpretive aspects of nonresponse: Nonresponse creates numerous analytic difficulties on major surveys. Minimizing this problem requires a greater understanding of the cognitive processes that lead respondents to decide not to answer surveys or particular survey questions. QDRL staff plan to conduct cognitive interviews using a variety of types of survey questions (behavioral and attitudinal) in order to explore these decision processes further. Survey nonresponse will be explored through examination of reasons that nonresponders provide for their unwillingness or inability to complete surveys. It is also possible that data will be collected through experimental questionnaires administered outside of the laboratory that explore the effect of various design decisions on item nonresponse. Contracts may be used for some components of this data collection and analysis.

Respondent Perceptions of Confidentiality and Survey Participation: To encourage participation, NCHS surveys such as the NHIS and NHANES depend on advance letters, promising confidentiality and explaining uses of the data collected. However, it is not known how well these statements are generally understood, and believed, by survey respondents. Therefore, QDRL staff proposes to conduct cognitive interviews of laboratory respondents in order to examine their comprehension of such statements. The results will be used to propose modifications to procedures used to communicate key issues related to informed consent, and to explain the need and purpose for survey data in a way intended to increase survey participation.

General Methodological Research: QDRL staff constantly evaluate and refine the cognitive interviewing methods used at NCHS, especially in order to respond to changes such as the wide-spread introductions of CAPI (Computer Assisted Personal Interviewing) and Audio Computer-Assisted Self-Interview (ACASI) as a data collection tools. In addition, QDRL staff may investigate the feasibility of using new technologies, such as video-over-internet conferencing software, such as Skype, GoToMeeting, Lync, or WebEx to conduct cognitive interviews, evaluate the quality of data obtained, and determine whether the mode of the interview impacts the interpretation of the survey questions. Further, QDRL staff regularly conduct applied research on questionnaire design issues, such as the optimal wording for measures of complex concepts related to health status, utilization, and behavior.

In 2015-2018 NCHS/QDRL staff plan to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, NCHS/ QDRL will work collaboratively with survey researchers from universities and other Federal agencies to define and examine several research areas, including, but not limited to: 1) differences between face-to-face, telephone, and video-over internet cognitive interviewing, 2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing, 3) reactions of both survey respondents and survey interviewers to the use of Computer Assisted Personal Interviewing (CAPI), Audio Computer-Assisted Self-Interview (ACASI), video-over internet and 4) social, cultural and linguistic factors in the question response process. Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, questionnaires that are of current interest (such as NHIS Modules) may be evaluated using several of the techniques described above. Or, different versions of a survey question will be developed, and the variants then administered to separate groups of respondents in order to study the cognitive processes that account for the differences in responses obtained across different versions.

These studies will be conducted either by QDRL staff, DHHS staff, or NCHS contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.

Research on human-computer interfaces/usability

The third major purpose of this data collection is to conduct research on computer-user interface designs for computer-assisted instruments, often referred to as “usability testing.” This research examines how survey questions, instructions, and supplemental information are presented on computer instruments (e.g., CAPI, Computer Assisted Self-Interviewing (CASI) instruments, ACASI, or web-based instruments) and investigates how their presentation affects the ability of users to effectively use and interact with these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions that are not to be read to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on. Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task.

Usability testing has many similarities to questionnaire-based cognitive research (described in Section 2.1), since it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat different, in that the typical user can be an interviewer (in the case of CAPI instruments) as well as a respondent (in the case of CASI/ACASI instruments). It also focuses more heavily on matters of formatting and presentation of information than traditional cognitive testing does.

Research Using Field Tests/Pilot Interviewing

The fourth major purpose of QDRL’s data collection is to apply unobtrusive field-based questionnaire evaluation techniques. The different questionnaires may be field tested/pilot-tested either individually or in groups, depending on developmental status of the instruments, the appropriateness of combining them, and their overall length. For most field tests, professional field interviews (such as Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) usually conduct approximately 200 pilot interviews (person/telephone). However, in 2012, a field test of almost 600 interviews, and a full scale dress rehearsal field test of 5,600 interviews was approved. Similarly, we anticipate that larger sample sizes may be required for future specific projects. There are four possible components to the proposed form of testing: a) a limited number of interviews on a draft version of the questionnaires are conducted using household respondents, b) a subset of interviews may be observed by NCHS and other staff trained in observational techniques, c) NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data and d) the potential for inclusion of built-in experiments i.e., two different versions of particular questions, in order to determine which version functions better in the field environment and mode of administration where test cases will be randomly assigned to receive questions in two different modes, i.e., CAPI or ACASI to assess the impact of mode on prevalence estimates and data quality.

Studies of the optimal design and presentation of graphical and textual material

The final major purpose is related to the growth of the Internet for collecting data (including Web-based surveys), and in disseminating health information. NCHS is the Federal government’s principal health statistics agency, and is responsible for collecting and disseminating many reports and volumes of data annually. During the last few years, the techniques developed for determining whether respondents understand survey questions have been applied with great utility to studying whether statistical publications and Web releases are optimally clear. One project, for example, involved the development and testing of a brochure designed by staff of the National Health and Nutrition Examination Survey (NHANES) to convert refusals to acceptance. Another recent project involved testing and evaluation of different Health Surveillance Map formats (choropleth versus isopleth) to determine if they affect ability to extract information from the maps for the Division of Adult and Community Health/the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP), CDC . We anticipate that there will be more work of this type during 2015-2018.

The major activities outlined above have well-demonstrated practical utility. As a result of laboratory testing, questionnaires may produce substantially less response error than would occur in the absence of this testing. Thus, users of NCHS data with measures that have been evaluated, in both Federal agencies and in the general health research community, will be less likely to be misled by erroneous statistical results. This assertion is supported by twenty four years of experience in using these techniques, and has been supported by findings presented at many statistical and research related conferences such as Joint Statistical Meetings (JSM), American Association for Public Opinion Research (AAPOR), American Sociological Association, and published in scientific journals such as “Applied Cognitive Psychology”, “Journal of Official Statistics”, “Public Opinion Quarterly”, “Field Methods”, and “Quality and Quantity”. The practical utility of field tests/pilot interviewing has also been supported in findings reported at an annual meeting of the American Statistical Association. Further evaluation of the efficacy of this method will be ongoing.

3. Use of Improved Information Technology and Burden Reduction

Usually, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), or web-based.

In addition, QDRL Staff use searches in Q-Bank to determine if a survey question has been cognitively tested. The regular use of Q-Bank reduces unnecessary testing as well as allows QDRL to build upon existing knowledge learned from past testing projects. Additionally, each cognitive interview is digitally recorded and stored on an internal, searchable video database. Like Q-Bank, this technology allows QDRL staff to build upon past projects and, at the same time, it improves the accountability of test findings.

4. Efforts to Identify Duplication and Use of Similar Information

The QDRL at NCHS is the only government facility that currently conducts testing and development of NCHS or other CDC questionnaires. Similar facilities at the Bureau of the Census and the Bureau of Labor Statistics bear the responsibility for testing survey questionnaires associated with their own agencies. The demand for QDRL activities exceeds available resources.

In order to identify duplication across federal agencies, QDRL hosts a publicly accessible online searchable database, Q-Bank, that contains all QDRL evaluation reports. QDRL encourages all agencies to submit their evaluation reports so that it is possible to track the work done across agencies as well as to build in existing knowledge.

5. Impact on Small Businesses and Other Small Entities

In the past, representatives of small businesses have been interviewed as part of testing of establishment surveys, such as the National Employer Health Insurance Survey (NEHIS) (OMB# 0920-0341). If such requests are made, these businesses will be approached in the same manner as the individuals we normally recruit; we will ask the organization to identify the appropriate staff members with whom to conduct the cognitive interviews.

6. Consequences of Collecting the Information Less Frequently

Individual projects usually involve one-time data collection activities. There are no legal obstacles to reducing the burden.

7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5

There are no special circumstances.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies

A 60-day Federal Register Notice for this collection was published on Month, March 17, 2015, (Vol. 80, No. 51, p. 13868-13869). The text of the notice is contained in Attachment B. No comments were received.

Consultants outside of CDC:

The following individuals have been consulted within the past year on survey methodology and/or on a specific project:

Gordon Willis

Cognitive Psychologist

Applied Research Program

Division of Cancer Control and Population Sciences

National Cancer Institute

6130 Executive Blvd, MSC 7344

EPN 4005

Bethesda, MD   20892-7344

[email protected]

Alisu Schua-Glusberg

Research Support Services

906 Ridge Avenue

Evanston, IL 60602

(847) 864-5677

[email protected]

Beth Ellen Pennel

Institute for Social Research

University of Michigan

426 Thompson Street

Ann Arbor, MI 48104-2321

(734) 647-2247

[email protected]

Consultants within CDC:

The following individuals have been consulted within the past year on survey methodology and/or on a specific project:

Eric Jamoom

Division of Health Care Statistics

3311 Toledo Road

Hyattsville, MD 20782

(301) 458-4798

[email protected]

Jim Dahlhamer

Division of Health Interview Statistics

3311 Toledo Road

Hyattsville, MD 20782

(301) 458-4403Phone

[email protected]

9. Explanation of Any Payment or Gift to Respondents

For most testing projects, cognitive interview respondents receive remuneration for several reasons:

 Typically, respondents are recruited for specific characteristics that are related to the subject matter of the survey (e.g., questions may be relevant only to people with certain health conditions). The more specific the subject matter, the more difficult it is to recruit eligible respondents. Remuneration helps to attract a greater number of potential respondents.

 Cognitive interviews require an unusual level of mental effort, as respondents are asked to explain their mental processes as they hear the question, discuss its meaning and any ambiguities, and describe why they answered the questions the way they did.

 They are usually asked to travel to the laboratory testing site, which involves transportation and parking expenses. (Many respondents incur additional expenses due to leaving their jobs during business hours, making arrangements for child care, etc.).

For a standard cognitive interviewing project in which one-hour interviews are conducted at NCHS and eligibility requirements are of average complexity, respondents will be given $40.00. Higher remunerations may be requested on a case-by-case basis for particularly difficult recruitments. For example, in a 2008 & 2009 study, the QDRL was unable to find epidemiologists willing to be interviewed for less than $75, and in 2011 & 2013 the QDRL was unable to find physicians willing to be interviewed for less than $100. On rare occasions, a lower remuneration is proposed.

It is important to offer remuneration sufficient to attract the full range of needed respondent types for cognitive interviewing projects. Inadequate respondent recruitment limits the effectiveness of the questionnaire evaluation. Requests and justification for remuneration will be included in each individual collection submission.

For activities that are meant to resemble the usual household interview (for example, field tests/pilot interviewing), respondents will not receive remuneration.

10. Assurances of Confidentiality Provided to Respondents

Confidentiality provided to respondents is assured by adherence to Section 308(d) of the Public Health Service Act (42 USC 242m) which states:

"No information, if an establishment or person supplying the information or described in it is identifiable, obtained in the course of activities undertaken or supported under section...306 (NCHS legislation),...may be used for any purpose other than the purpose for which it was supplied unless such establishment or person has consented (as determined under regulations of the Secretary) to its use for such other purpose and (1) in the case of information obtained in the course of health statistical or epidemiological activities under section...306, such information may not be published or released in other form if the particular establishment or person supplying the information or described in it is identifiable unless such establishment or person has consented (as determined under regulations of the Secretary) to its publication or release in other form,..."

In addition, legislation covering confidentiality is provided according to section 513 of the Confidential Information Protection and Statistical Efficiency Act (PL 107-347) which states:

“Whoever, being an officer, employee, or agent of an agency acquiring information for exclusively statistical purposes, having taken and subscribed the oath of office, or having sworn to observe the limitations imposed by section 512, comes into possession of such information by reason of his or her being an officer, employee, or agent and, knowing that the disclosure of the specific information is prohibited under the provisions of this title, willfully discloses the information in any manner to a person or agency not entitled to receive it, shall be guilty of a class E felony and imprisoned for not more than 5 years, or fined not more than $250,000, or both.”

All NCHS/QDRL staff, as well as QDRL contractors receive appropriate annual privacy and confidentiality training, and sign a yearly Nondisclosure Affidavit (see Attachment C). Staff from collaborating agencies are also required to become designated agents and sign the nondisclosure statement.

10.1 Privacy Impact Assessment Information

The NCHS Privacy Act Coordinator has reviewed this request and has determined that the Privacy Act is applicable. The related System of Records Notice is 09-20-0164 Health and Demographic Surveys Conducted in Probability Samples of the U.S. Population.

A Privacy Impact Assessment was submitted on July 25, 2013. The QDRL continues to collect, on a confidential basis, data needed in order to conduct QDRL studies. The process of informing respondents of the procedures used to keep information confidential begins with the telephone screener and will carry through to the interviewer and all communications with potential respondents. Materials will include all elements of informed consent, including the purpose of the data collection, the voluntary nature of the study, audio or video recording of the interview, and the effect upon the respondent for terminating the interview at any time.

Data in identifiable form is collected for linkage of various QDRL forms (informed consent documentation and respondent demographics) and audio and video recordings. The QDRL also uses some identifiable data (name, phone number, email address) to contact previous respondents for QDRL studies. The ability to match respondents to other data (informed consent documents, respondent demographics, and audio/video recordings) greatly expands the usefulness of the data at a very low cost.

Only those QDRL staff, specially designated agents as outlined in the informed consent form, and on occasion QDRL contractors, who must use the personal information for a specific statistical purpose, can use such data.

The collection of information in identifiable form requires strong measures to ensure that private information is not disclosed in a breach of confidentiality. Storage of confidential data is protected through procedures such as an internal QDRL LAN, passwords and carefully restricted access.

Informed Consent and Voluntary Nature

QDRL respondents/interviews conducted at NCHS

Respondents are recruited through media advertisements, flyers, and word-of-mouth, and either call the QDRL voice mail system or contact a person coordinating the recruitment. Data collection for this project is authorized under 42 USC 242k (Section 306 of the Public Health Service Act).

During the telephone screener (Attachment D), potential respondents are informed that answering the telephone screener questions to determine their eligibility for the study is completely voluntary. They are informed that we are required by law to use the information they provided in the telephone screener for statistical research only and to keep it confidential, and that the law prohibits us from giving anyone any information that may identify them without their consent. In addition, respondents who are determined to be eligible for the study are informed during the telephone screener that the information they provide during the cognitive interview is confidential.

Prior to the start of the cognitive interview, QDRL respondents read and sign Attachment E, Informed Consent Form (written at an 8^th grade reading level). There are five templates in the attachment to cover various consent situations. The consent form states that participation is voluntary, they are free to terminate the interview at any time, and if they do so, they will still receive remuneration. The consent form describes the purpose of the interview and recording, specifies that the recordings may be played for other staff working closely on that project, that voice and face identifiers will remain on the recording, and that they may be recognized by a staff member viewing or listening to the recording. Respondents are given a copy of the consent form, which contains contact information for the QDRL Laboratory Manager, the NCHS Research Ethics Review Board (ERB), and the NCHS Confidentiality Officer.

At the close of the cognitive interview, a respondent may also be asked by the interviewer to sign Attachment F, the Special Consent for Expanded Use of Video and Audio Recordings Form. The purpose of this form is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); recordings of interviews with minors will never be shown to others not included in the study staff. Respondents are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer. If respondents grant Special Consent, recordings are kept for as long as there is a justifiable, scientific use for the recordings as determined by the NCHS Research Ethics Review Board.

Additionally, at the close of the interview the respondent may also be asked by the interviewer to sign Attachment G, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL respondents’ privacy and the recording. This form is not used in the case of interviews with minors (persons under the age of 18); recordings of interview with minors will never be shown to others not included in the study staff. Respondents are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer.

QDRL respondents/interviews conducted off-site¹: Sometimes interviewers must travel to conduct cognitive interviews In these cases, a mutually agreeable location will be chosen. In all cases, extreme care is taken with audio and video recordings and any materials that contain personal identifiers such as the Informed Consent Form, Special Consent for Expanded Use of Video and Audio Recordings, or the Special Consent to Send Video and Audio Recordings to Off-site Researchers. Materials are then transported to the QDRL, where standard procedures are followed.

Focus groups: In focus group settings, participants are together and obviously can hear each other’s comments, statements, and questions. Participants are told in their initial telephone screening interview that they will be participating in a discussion group with other volunteers. Before the group discussion begins, participants sign the Informed Consent Form (Attachment E) which is tailored to specify that they will be participating in a focus group. The Informed Consent also states that they will be asked to pick a name and put it on a name tag, and that they do not have to use their real name. It is the responsibility of the interviewer (usually referred to as a moderator when conducting a focus group) to instruct the group that the information discussed will be held confidential by NCHS staff and should be treated confidentially by all respondents. Participants are strongly urged to respect the privacy of the other respondents and not to discuss with others what was discussed by the group.

At the close of the focus group, participants may be asked by the moderator to sign Attachment H, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Respondents of Discussion Groups Form. The purpose of this form is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from a focus group. Use of this form is at the discretion of the moderator and is typically warranted if (1) the focus group demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used in the case of focus groups with minors (persons under the age of 18); recordings of focus groups with minors will never be shown to others not included in the study staff. Participants are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS Research Ethics Review Board Chair, and the NCHS Confidentiality Officer. If participants grant Special Consent, recordings are kept for as long as there is a justifiable, scientific use for the recordings as determined by the NCHS Research Ethics Review Board. If any one respondent from the focus group does not grant special consent, the recording will not be used in this way.

Contractor conducted interviews

On the rare occasion when contractors are used to collect data as part of QDRL projects, they are contractually bound by NCHS confidentiality provisions and must submit documentation concerning ­their safeguarding practices to NCHS prior to data collection. This is standard NCHS practice and does not reflect a special QDRL procedure. The contractor employee will view the NCHS confidentiality video and sign the NCHS non-disclosure statement before starting work on the project.

Field Tests/Pilot Interviews

For field test/pilot interviews of household and telephone respondents, standard operating procedures regarding informed consent and survey administration procedures specific to the survey being tested will be followed.

Security

Confidentiality of responses and safeguarding of data at NCHS

The QDRL has a routine set of measures to safeguard confidentiality, including the following: all QDRL staff who have access to confidential information are given instruction by the QDRL Laboratory Manager on the requirement to protect confidentiality and are required to sign a pledge to maintain confidentiality; only such authorized QDRL personnel are allowed access to confidential records and only when their work requires it; when cognitive interviews or focus groups are conducted off-site, data are secured to insure that there is no loss in transit; and when confidential information is not in use, it is stored in secure conditions.

All respondents receive a copy of Attachment E, Informed Consent Form, which describes the procedures by which confidentiality of data identifying individuals is maintained.

Informed Consent documents

Informed consent documents are stored by project in a separate drawer in a locked filing cabinet in the locked office of the QDRL Recruiter until they can be electronically scanned and moved to the secure QDRL Local Area Network (LAN). No one other than the QDRL Laboratory Manager or the QDRL Recruiter have access to the informed consent forms.

QDRL Respondent Database: A custom-designed QDRL Respondent Database contains personal identifiable information and demographic information on respondents who have participated in past QDRL studies such as name, phone number, email address, age, marital status, ethnicity, race, education, employment status, and household income. The QDRL Respondent Database is used to conduct computerized searches to locate computer records of past respondents having salient characteristics for use in future studies. The QDRL Respondent Database is also used to produce periodic tabulations and reports on database characteristics and response rates. The QDRL Respondent Database is housed on the secure QDRL LAN. The QDRL LAN is located in the QDRL Control Room. Because the QDRL LAN is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS.

Safeguarding of audio and video recordings: The QDRL Recruiter/QDRL Staff label each audio and video recording by a unique respondent identifier number, date, time, and project title. No other identifying information is labeled on the recording. Recordings are viewed/listened to in the Observation Room, or on QDRL Staff’s desk top computers which are hardwired to the secure QDRL LAN. Only QDRL Staff holding proper passwords and keys to staff locked offices have access to interview recordings on the QDRL LAN. The QDRL LAN is located in the QDRL Control Room and locked when not in use. Because the server is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS. Recordings are stored in the Observation Room which is locked when not in use, or on the secure QDRL LAN.

Safeguarding of video recordings viewed at locations other than NCHS: Depending on the project, sponsors and collaborators may be from CDC, and occasionally from other DHHS or outside Federal agencies. The Informed Consent Form is tailored to describe each project and will specify which agencies are collaborating in the research and which staff(s) may be viewing the recording. Any outside NCHS collaborator viewing the recording (whether onsite at NCHS or off-site at a collaborating agency) will be required to sign a Nondisclosure Affidavit (Attachment C). Occasionally, a collaborator will be unable to travel to NCHS to view the recording. For those recordings in which Special Consent to Send Video and Audio Recordings to Off-site Researchers (Attachment G) were granted, a contract will be developed and signed by the Director of NCHS and the Director of the relevant organization. The contract will state how they will protect the respondents’ privacy and the recording until its return to NCHS. The contract will be coordinated through the NCHS Confidentiality Officer who will oversee shipment and the return of the recording to NCHS.

Reports and publications: No respondent names or other personal identifying information is included in any reports or publications of cognitive testing results.

Presentations: No respondent names or other non-photographic identifying information is included in any presentations of cognitive testing results. As outlined in the standard informed consent and the special consent for expanded use of video and audio recordings, QDRL respondents have been informed that voice and face identifiers will remain on the recording and have granted permission for the audio or video recordings to be played either to individuals working closely on the project or at conferences, meetings, or in the classroom.

Protocol #2010-19 Laboratory Based Questionnaire Design (QDRL) was approved by the NCHS Research Ethics Review Board on September 24, 2014 (Attachment I).

11. Justification for Sensitive Questions

Most of the questionnaires currently proposed for study generally do not contain questions that are highly sensitive in nature. There are some exceptions, such as the National Survey of Family Growth, NHIS questions on income and HIV, and NHANES questions on sexual behavior. Again, one purpose of pre-testing such questions is to determine means for fashioning these questions in such a way that sensitivity is minimized, and responses are valid.

12. Estimates of Annualized Burden hours and costs:

1. An average of 8,000 individuals (24,000 over the three year period) participate in QDRL activities in a given year and the average annual respondent burden is estimated to be 4,383 hours. For the full three years there will be an estimated 13,150 hours of burden. Annualized estimates of respondent burden for each of the questionnaire development studies, over the course of data collection, are provided below. For most questionnaire development studies it is anticipated that interviews will last one hour. For some questionnaire development studies, questionnaire administration is anticipated to frequently require less than an hour of a respondent’s time (for example, a fifteen-minute interview may be conducted), and in rare cases, the burden may be more than one hour. Because the hours per response in questionnaire development studies are expected to vary, we will select the final sample size for each project in such a way that the total burden hours do not exceed the estimate listed above. For focus groups, the usual amount of time is 90 minutes (1.5 hours) which includes instructions and ancillary paperwork.

For interviews in the laboratory, time required to travel to the lab is not covered, because distances and modes of transportation are unknown. No retrieval of information by respondents is anticipated; although it is possible that validation of data at some point may require respondents to check records, probably those kept at home. In that case, the study will be designed so that the response time includes record retrieval. All estimates are based on NCHS' past experience (1988 through 2014).

Estimated Annualized Burden Table

Types of Respondents	Form Name	Number of Respondents	Number of Responses per Respondent	Average hours per response (in hours)	Total Burden Hours
Individuals or households	Eligibility Screeners	4,000	1	5/60	333
Individuals or households	Developmental Questionnaires	3,900	1	1	3,900
Individuals or households	Focus Group Documents	100	1	1.5	150
Total		8,000			4,383

Estimated Annualized Burden Costs to Respondents.

The average annual response burden cost for the QDRL is estimated to be $181,680. The hourly wage estimate is based on the Bureau of Labor Statistics May 2014 National Occupational Employment and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm). There is no cost to respondents other than their time to participate.

Type of Respondent	Form Name	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Individuals or households	Eligibility Screeners	4,000	$22.71	$90,840.00
Individuals or households	Developmental Questionnaires	3,900	$22.71	$88,569.00
Individuals or households	Focus Group Documents	100	$22.71	$2,271.00
Total				$181,680.00

13. Estimates of Other Total Annual Cost Burden to Respondents and Record keepers

None.

14. Annualized Costs to the Federal Government

The cost to the government consists mainly of the salaries of the QDRL staff that will (1) assist the questionnaire designers in the design of appropriate laboratory instruments, (2) recruit, schedule, and assist in interviewing volunteer respondents, and (3) assist in the analysis of the results and recommend changes in questionnaire wording.

Total annualized project costs are as follows:

NCHS costs for QDRL staff to plan, conduct, and analyze the outcomes of the questionnaire development activities:

Managerial 1.00 FTE $141,660.00

Professional 7.00 FTE $558,980.00

Support 1.00 FTE $98,916.00

Remuneration of QDRL respondents 4420 @ $40 $176,800.00

QDRL Contract Staff (including remuneration) $712,977.00

Contracts for assistance with

methodological research $30,000.00

Off-site travel: (see note below under

travel costs) $10,000.00

Materials for conducting household

interviews $500.00

Flyers $200.00

Advertisements $21,000.00

Hardware and software upgrades $50,000.00

Annual Total $1,801,033.00

3 Year Total (for generic submission) $5,403,099.00

Travel costs: Most data will be collected in NCHS office space. However, it will be more efficient in certain instances to hold interviews with individuals at other locations, which will involve some travel costs. Further, household interviews will require limited numbers of in-person interviews in respondent households. Household interviews will be done locally, in order to limit travel costs, unless there is a compelling reason to do otherwise (for example, if respondents critical to the study can be interviewed only at a distant location).

15. Explanation for Program Changes or Adjustments

This is a generic clearance. The current burden is 9,450 total hours for three years (which would have been 3,150 annualized hours). We are requesting 4,383 annualized hours (13,150 burden hours over three years), which is an increase of 1,233 hours per year (for a total increase of 3,699 hours over three years). The difference is due to an anticipated increase in the number and size of projects being undertaken.

16. Plans for Tabulation and Publication and Project Time Schedule

This clear­ance request is for questionnaire development activities to be conducted prior to survey production and for developmental work that will guide future questionnaire design. The majority of laboratory investigations will be analyzed qualitatively. The survey designers and lab staff serve as interviewers and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. Final reports will be written that document how the question performed in the interviews, including question problems as well as the phenomena captured by the survey question. All reports will be placed on Q-Bank for public access. Reports are used to provide necessary information to guide designs for redesigning a question prior to fielding as well as to assist end users when analyzing the survey data. For field tests/pilot interviewing activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews in order to determine where additional problems occur. Because NCHS is using state-of-the-art questionnaire development techniques, methodological papers will be written which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the cognitive interview and observation techniques.

Each individually submitted information collection will include a project time schedule specific to that project.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

Not applicable.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

Not applicable.

1Off-site interviews fall into two categories. First, it is not always feasible for individuals to travel to the QDRL, or it may be more efficient for interviewers to travel to a particular site. Second, we occasionally conduct establishment studies where a visit to the business location is pertinent to data collection.

24