NCCDPHP GCT SSB Final

Information Collection Request

NEW

Generic ICR for Cognitive Testing and Pilot Testing

For the National Center for Chronic Disease Prevention

and Health Promotion

Supporting Statement: Part B

Program Official/ Contact

Carol Pierannunzi, PhD

Division of Population Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Atlanta, Georgia

Phone 770-488-4609

Fax 770-488-5965

Email: [email protected]

November 25, 2019

List of Attachments:

1. Attachment 1: Authorizing Legislation

2. Attachment 2: Federal Register Notice

3. Attachment 3: Example of Questionnaire Proposal Process from the BRFSS

4. Attachment 4: List of Divisions Within the NCCDPHP

5. Attachment 5: Example of Vendor Cognitive Testing Report

6. Attachment 6: Example of Screener for Consent to Participant in Cognitive Testing

7. Attachment 7: Multimode Pilot Final Report

8. Attachment 8: Response to Public Comments

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The purpose of collections under this generic clearance is to allow information collection processes currently in place in the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) to conduct cognitive testing and pilot testing of methodological procedures. These procedures include testing of new and existing questions, question formatting, survey protocols, data collection software systems and other related processes in order to identify problems and improve efficiencies in public health surveillance. Each submission under this generic clearance will include specific information as to the respondents and sampling to be undertaken for the cognitive and/or field test specified. When international populations or minor children are part of the pilots, methods will include specific steps to ensure that informed consent is obtained. The NCCDPHP data collection activities include the Behavioral Risk Factor Surveillance System (OMB No. 0920-1061; Expires 3/31/2018), which conducts approximately 3-4 cognitive testing reviews and one pilot test annually. In addition, this clearance will cover a number of other cognitive testing and pilot testing activities, the scope of which is not known at this time, for other surveys housed within the NCCDPHP. Surveys conducted by NCCDPHP divisions other than the Division of Population Health include, but are not limited to, the Pregnancy Risk Assessment Monitoring System (PRAMS), the Youth Risk Behavior Survey (YRBS) and various tobacco surveys conducted by the Office on Smoking and Health. Therefore, the total number of respondents has been adjusted to allow which may be created by changes in existing surveys, unanticipated needs of changing public health conditions or other contingencies.

The table below illustrates the estimated respondent universe and burden hours under this generic clearance proposal. Since this is a new generic request, the estimates presented in Table 1 are based principally on experience with the Behavioral Risk Factor Surveillance System (BRFSS). However, some of the modes listed in the table are not part of the methods used for the BRFSS, therefore the distribution with types of tests may be adjusted across categories as specific testing needs arise.

Table 1 Respondent Burden by Type of Information Collected
Type of Respondents	Type of Information Collection	Estimated number of ICs over 3 years	Number of respondents per IC	Total respondents over 3 years	Annualized number of respondents	Average burden per response (in hours)	Total annualized burden hours
Adults ≥ 18 years of age	Screening for cognitive testing	15	500	7500	2500	15/60	625
	Screening for field testing	6	1200	7200	2400	15/60	600
	Cognitive testing in person	15	300	4500	1500	60/60	1500
	Cognitive testing by phone	15	300	4500	1500	45/60	1125
	Cognitive testing by ABS/mail/web	6	400	2400	600	30/60	300
	Pilot testing in person	15	300	4500	1500	45/60	1125
	Pilot testing by phone	15	300	4500	1500	45/60	1125
	Pilot testing by ABS/mail/web	15	300	4500	1500	45/60	1125

Cognitive testing participants for in-person interviews will be recruited by advertisement or through established methods for recruiting hard-to-reach participants. For example, networks of persons who are known to have quit smoking may be asked to review questions on smoking cessation. Respondents participating by telephone will be sampled using Random Digit Dialing (RDD) methods and participants who take part in mail format testing will be selecting using Address Based System (ABS) samples. Each request submitted under this generic clearance will include specifics as to how samples will be selected and how respondents will be informed as to the voluntary nature of their cooperation in the cognitive and/or field test. An example of a screening script for cognitive testing by phone is found in Attachment 6.

2. Information Collection Procedures

As with all other aspects covered by this generic clearance, information collection procedures specific to each cognitive testing and field testing activity will be detailed with each submission.

Cognitive Testing

Cognitive testing is conducted early in the process of considering questions for adoption. Questions which are undergoing cognitive testing may have not been used in previous surveys due to the emergence of a new public health concern (such as e-cigarettes) or as a result of their inclusion in a survey that is being offered in a different mode. Cognitive testing is conducted in a controlled setting, generally in an office setting. Participants may be recruited to take part in cognitive testing through online or newspaper advertisements. If the participants are not recruited to be present at a physical location, they may be called and recruited by phone. Recruited participants are asked to review questions and/or surveys to discuss their impressions of the items under consideration. Participants may be asked to identify their impression of the questions, the response set, individual words within the question, or the focus of the questionnaire itself. Participants may respond individually or as members of focus groups.

The questions under consideration may be presented to the participants in different formats (online, on paper, verbally or using mobile devices). Individual respondents may be asked to interpret the meaning or ease of answering questions based on formats or order of the questions/ responses (see Attachment 5 for an example of questions asked in cognitive testing). Groups of individuals may also discuss what they found to be confusing or visually appealing about questionnaires that are presented online or on paper. A single respondent may review the same question in a number of formats to rate or rank and provide information on the favorable aspects of each version.

Cognitive testing is an iterative process of adjustment of a question, a survey mode or a format, until it optimizing the respondents’ ability to understand the concept of the information sought by the researcher and produce a valid response.

Pilot Testing

Pilot tests are used to determine whether a new approach or data collection methodology may be useful are a part of regular IC in the near future. Pilot tests may be proposed as a potential solution to a data collection issue (such as nonresponse, targeting a specific hard-to-reach population, or determination of the effectiveness of a new calling protocol). Pilot tests do not impact or take the place of regular data collection efforts at the time of their implementation. For the BRFSS, for example, pilot tests may be conducted in a selected number of states without changes in the regular data collection processes of those states. Pilot testing may include testing groups of individuals in order to compare rates of response when differing questionnaire formats or modes of response are provided. Pilot testing may also be used to determine mode effects on a single question, whether and how temporal reminders affect response, bias effects of formats and issues related to sensitive questions. Feasibility tests of new modes of data collection are of unique importance as new methods of contact are under review to alleviate issues related to response rate decline.

Pilot testing results are often used to inform future survey construction, sampling, methodologies, interpretation of responses and may often be used for publication and presentation to survey and surveillance methodological audiences. Participation by the public in pilot testing is voluntary and responses to questions are confidential.

3. Methods to Maximize Participation Rates

A primary effect of cognitive testing and pilot testing is to maintain high levels of participation and data quality in the public health surveillance process throughout the NCCDPHP. Participation in cognitive testing and pilot testing is voluntary, but respondents will be encouraged to participate by explanations of the need for their input in the introduction of each survey. Persons participating in the in-person phase of cognitive testing may receive incentives for participation based on their time and travel. It is standard industry practice to provide compensation for participation in these types of activities in order to achieve an appropriate sample.

4. Tests of Procedures

Prior to cognitive and field testing, information may be solicited from data user partners (state health departments, CDC program offices) as to potential problems with questions or methods which will be tested. In addition, if new questions or procedures are being considered for use of any NCCDPHP surveys, any available information on reliability and validity of the questions or methods to be tested will be obtained. Often proposals for questions or methods come from CDC programs or other federal agencies which supply information regarding how questions or versions of questions which are proposed for use on NCCDPHP surveys have been used in the past. This information is submitted as part of the question proposal process (see Attachment 3 for an example of a proposal process). Pilot testing may be proposed as a potential solution to an issue encountered during data collection, such as declining response rates, nonresponse bias due to noncoverage of hard-to-reach populations, increased costs of information collection methods used in the past or other issues. Small scale testing of new methods using pilot tests may ascertain whether changes in methods can offer IC protocols which can provide quality data with greater efficiency.

Data collected under this generic will be examined for patterns of consistent responses among recruited subject, differences by demographic groups and/or numbers of persons who are unable to answer questions as worded. Paradata on the number and characteristics of persons who participate by mode (telephone, email, in-person or mail) will also be analyzed to determine whether mode of data collection introduces bias into results. Paradata on the efficiencies of samples (such as the number of contact attempts or interviewer effort measures) will be examined by mode and type of administration of survey questions. Efficiencies in data collection method will be compared by costs per completed interview, nonresponse bias estimation and/or response rates.

Most analyses will use nonparametric measures of relationships between respondent characteristics, cost data and measures of response, bias and participation. Attachment 5 provides an example of final reporting of data for a cognitive test including analyses and Attachment 7 provides examples of analyses of data from pilot testing.

5. Statistical Consultants

Whenever it is methodologically sound and appropriate to do so, the NCCDPHP may obtain technical assistance from experts within other federal agencies, state health departments and/or outside the federal government to implement cognitive and/or field testing procedures. In some instances, statistical expertise will be available from among NCCDPHP partners or contractors.

6