OMB Memo

Crowdsourcing Cognitive Interviews

Submitted Under Generic Clearance for Questionnaire Pretesting

Request: The Census Bureau plans to conduct additional research under the generic clearance for Questionnaire Pretesting (OMB #0607-0725). We will conduct online pretesting of select questions from the National Survey of Children’s Health (NSCH). We are targeting specific questions previously tested in two rounds of in-person cognitive interviews conducted under the generic clearance for questionnaire pretesting (OMB #0607-0725).

Purpose: The purpose of this research is to investigate the utility and potential limitations of using crowdsourcing for pretesting, relative to in-person cognitive interviews. Crowdsourcing could potentially be used when in-person cognitive testing cannot be done, or to confirm findings from previous in-person rounds of testing. Researchers from the Survey Methodology (SM) area of the Demographic Statistical Methods Division will retest the results of in-person NSCH cognitive interviews using Qualtrics and compare the results to those obtained through prior in-person interviews to see if similar conclusions would be drawn.

Population of Interest: Residents of the United States and its territories with children aged 0-17 living in the participant’s household.

Timeline: We intend to conduct the online pretesting surveys in June 2020. The survey will be open until we have received 250 completed responses.

Sample We will request no more than 250 completed responses using a Qualtrics panel. We require a general population sample over 18 years old with age and regional quotas. Qualtrics will select the participants according to our requirements. Qualtrics will recruit English speaking participants, with at least one child living in the household (age 0-17), evenly divided across the Census regions (Figure 1), with a minimum of 25 participants from each of the following age groups: 18-29, 30-39, 40-49, 50-59, and 60+. Qualtrics will recruit a maximum of 50 participants from the 60+ age group.

Figure 1

Recruitment: Qualtrics will recruit from their panel as needed to achieve the sample targets.

Survey Administration: The questionnaire will be administered online using the survey platform Qualtrics. Respondents will receive an invitation with a link to the survey, which will then take them to the Qualtrics instrument for this study.

Questionnaire: Participants will be asked to self-sample a child from their household and answer questions about this child’s health and health care. They will be assigned to one of two questionnaires based on the age of the sample child. Each respondent will be asked a subset of questions from the NSCH concerning the sample child’s health, household, and the child’s caregivers. Participants will also be asked follow-up probes about their thought processes, interpretations of specific terminology, and their perceptions of specific survey elements.

Informed Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project.

Incentive: Participants from the Qualtrics sample will be given an incentive by the Qualtrics vendor. Per Qualtrics, the exact amount and form that each respondent receives can vary depending on the participants’ profile, how they were recruited, and the form of incentives they have elected to receive (i.e. e-gift cards, points, cash, etc.). Respondents typically receive ~$1.00-$1.50 or equivalent point value for completing a 15 to 20-minute consumer survey. 

Length of Interview: We estimate that the survey will take an average of 15 minutes for each complete response. We will allow up to 250 complete responses for a total of about 62.5 hours. Each email will take about 1 minute to read. The total estimated burden of this research is 66.5 hours.

Table 1. Total Estimated Burden

Research Activity	No. of Respondents	Participation Time	Burden
Reading email invitations	250	1 minute	4 hours
Survey	250	15 minutes	62.5 hours
Totals			66.5 hours

Below is a list of materials to be used in this research.

1. Survey Content for Participants with a Child Aged 0-5

2. Survey Content for Participants with a Child Aged 6-17

3. Qualtrics Survey Screenshot Examples

The contact person for questions regarding data collection and the design of this research is listed below:

Rachel Horwitz

Demographic Statistical Methods Division

U.S. Census Bureau

(301) 763-2835

[email protected]

Page 2 of 2