Justification

Volume I

National Household Education Surveys Program (NHES)

Screener Opt-out Cognitive Interviews

OMB# 1850-0803 v.231

April 2018

National Center for Education Statistics (NCES)

Attachments:

Attachment 1: Communication materials (Recruitment emails, advertisements, screeners, and consent forms)

Attachments 2 and 3:

Attachment 2: Study materials (Cognitive interview protocol, NHES opt-out screener cover letter for testing)

Attachment 3: Opt-out Screener Surveys (NHES opt-out screener survey versions to be tested)

Justification

The National Household Education Survey (NHES) is conducted by the National Center for Education Statistics (NCES) and provides descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which second-stage “topical” survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

In NHES:2016 and in the NHES 2017 web-pilot, the screener enumerated the entire household including adults and children because the Adult Training and Education Survey (ATES) was among the second-stage surveys fielded. NHES:2019 will not field ATES. NHES:2019 will include only the Early Childhood Program Participation (ECPP) survey and the Parent and Family Involvement in Education (PFI) survey. Therefore, the NHES:2019 screener was returned back to the NHES:2012 screener to only ask to enumerate the children in the household. A concern for NHES:2019 is that only enumerating children in the screener could potentially result in nonresponse from households without children that elect not to complete the screener because they may feel it is not applicable. Response from households without children is important to the study because it provides data necessary to calculate accurate prevalence rates and because it indicates which households are or are not eligible for a topical survey.

For NHES:2019, NCES plans to experiment with the use of an “opt-out” paper screener and cover letter which allows households without children (about 60 percent of all sampled households) to indicate on the front cover of the screener that there are no children in the household without having to open the survey to complete their response. Our goal is to increase the response rate to the paper screener by making it easier for households without children to quickly realize the low burden required to complete the instrument and thus for these households to respond.

In preparation for the NHES:2019 administration, this request is to conduct cognitive interviews in English and Spanish to test this adapted version of the screener. These interviews will include testing three different formats of the screener questionnaire: 1) a longer version that includes all the instructions and wording that are part of the original screener on the cover, 2) a shorter version that highlights the question on the cover as to whether or not the household has children, and 3) a screener that emphasizes that if the household has no children they only need to answer this one question and return the survey. The request to conduct the recruitment portion of this study was approved on May 1, 2018 (OMB#1850-0803 v.230). This request is to conduct the cognitive interviews. This submission provides the study’s data collection materials, including:

• Cognitive interviews recruitment materials – Attachment 1

• Cognitive interview protocol (in English and Spanish) – Attachment 2

• NHES opt-out cover letter (bilingual) – Attachment 2

• NHES opt-out screener questionnaires (3 versions, each in English and Spanish) – Attachment 3

The primary deliverable from this cognitive interviews study will be an English and Spanish version of the NHES:2019 opt-out screener for use in NHES:2019, along with a report summarizing findings from this study (both of which, in turn, will be added by June 18, 2018, to the NHES:2019 regular clearance materials that are currently in a 60-day public comment period).

Design

Cognitive Interviews

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just responded to. Techniques include asking probing questions to clarify points that are not evident from the think-aloud comments, and responding to scenarios. In these interviews, probes will be used to:

• verify respondents’ interpretation of the screener questions and instructions;

• check respondents’ understanding of the meaning of specific terms or phrases used in the questions and instructions; and

• explore respondents’ reaction to different formats and constructions of the opt-out screener.

The cognitive interviews study will be iterative, in that question wording, format design, and interview protocol may change during the testing period in response to problems identified during earlier interviews. Thus, throughout the cognitive interviews study, we might test multiple revisions of each version of the screener instrument. The most effective version of the instrument will be chosen for NHES:2019.

To adequately test the survey, it is necessary to distribute the cognitive interviews across participants with and without children and across education levels. We propose to conduct up to 25 interviews, at least 8 of which will be with Spanish speakers.

Interviews will be audio-recorded.

Recruiting and Paying Respondents

Participants will be recruited by the American Institutes for Research (AIR) using multiple sources, including Craigslist ads, and personal and professional contacts. An example recruitment advertisement and e-mail are included in Attachment 1. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. Interested individuals will be asked to complete an online or phone screening questionnaire (Attachment 1). Most of the 25 interviews will take place in the AIR offices in the DC-Metro area, with a few interviews to occur in quiet, public places, such as a library or community center, in the Grand Rapids, Michigan and Jacksonville, Florida areas, or other areas near AIR in Austin, Texas; San Mateo, California; and Chicago, Illinois. To assure that participants agree to take part in the interviews and to thank them for their time and for completing the interview, each respondent will be offered $40.

Assurance of Confidentiality

The statement below will be presented in all written materials (e.g., letters, emails) and read at the start of the cognitive interview. Participants will also be informed that they can stop the interview at any time.

NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your participation is voluntary and all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

All contractor staff working on the cognitive interviews will sign NCES Affidavit of Nondisclosure. Personal information (e.g., name, address) will be collected for recruitment purposes, but on the data file used for analyses, respondents will be identified only by a unique study ID number assigned to each participant. Within 48 hours of respondents’ participation in the interview, the discussion notes will be edited, organized, and cleaned, and all identifiers will be stripped from the data set. All computer files will be password-protected and hard copies will be locked in secure locations (e.g., data will be in locked file cabinets within locked offices). Only contract staff working directly on the data analysis portion of the project will have access to the data files. Once the final report is created, all personally identifiable information will be destroyed. All presentations of data in reports will be in aggregate form, with no links to individuals.

A consent form will be collected for all participants in the cognitive interviews. The consent form is included in attachment 1.

Estimate of Hour Burden

We expect the interviews to be approximately 30 minutes in length. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews per eligible participant (thus an estimated 300 screenings to yield 25 participants). This will result in an estimated total of 28 hours of respondent burden for this study.

Table 1. Estimated response burden for NHES:2019 screener opt-out cognitive interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	300	300	0.05	15
Cognitive Interviews	25*	25	0.50	13
Total	300	325	-	28

*Subset of initial contact group, not double counted in the total number of respondents.

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

Study recruitment will begin on May 2, 2018. Interviewing is expected to be completed within one month of OMB approval. After the interviews are completed, screener items and format will be finalized and will be added by June 18, 2018 to the NHES:2019 clearance materials that are currently in a 60-day public comment period.

Cost to the Federal Government

The cost to the federal government for this study is approximately $41,864 (already accounted for in the request to recruit participants for this study; OMB# 1850-0803 v.231).