Justification

Volume I

2016 National Household Education Surveys Program (NHES)

Phase 3 Cognitive Interviews for English and Spanish Recruitment Materials

OMB# 1850-0803 v.136

April 2015

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. Such topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, attainment of non-degree credentials, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. In 2016, the NHES will again field the PFI and the ECPP, plus the Adult Training and Education Survey (ATES). This will be a two-stage mail study. In the first stage, households will be screened to determine if they contain eligible members. If eligible members are in the household, within household sampling will be performed. Finally, topical surveys will be administered to the selected household members. Out-of-high-school adults ages 16 to 65 will be eligible to receive the ATES.

In order to reduce the respondent’s cognitive and time burden and to accommodate the materials development schedule, we have divided testing of Spanish NHES survey materials into three phases. This request is to conduct the phase 3 cognitive interviews to test and revise the language and translation of contact materials for a web experiment planned for NHES:2016. Because the web experiment contact materials are new to NHES in both English and Spanish, we will conduct English and Spanish interviews. Phase 1 cognitive interviews tested the language and translation of Spanish letters, postcards, other contact materials, screener instruments, and a few items from the ATES that have not been translated previously. Phase 2 focused on the Spanish topical questionnaires.

We will test only one version of each type of communications material because the language used in initial letter and nonresponse follow-up communications are very similar to each other. We will test:

• Initial web screener mailing,

• Second mailing for ECPP topical survey, where screener respondent is someone different from topical respondent, and

• Initial mailing for ATES topical survey, where screener respondent is the same as topical respondent.

Cognitive testing has been used for other NHES surveys in past years. The objective of this round of cognitive interviews is to identify and correct problems of ambiguity or misunderstanding in respondent contact materials and in their translations to Spanish. This should result in a set of contact and data collection materials that are easier to understand and therefore less burdensome for respondents.

Recruitment materials and participant consent forms for cognitive interviews are provided in attachment 1, the recruitment screener in attachment 2, the cognitive interview protocol in attachment 3, and the NHES:2016 respondent contact materials to be tested in attachment 4.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to answer a series of questions about the materials they have just interacted with. Techniques include asking probing questions, as necessary, clarifying points, and responding to scenarios. Interviews are expected to last about 1 hour and will be conducted by trained bilingual (Spanish/English) cognitive interviewers and, depending on the respondent, in English or Spanish. It is expected that the instruments and interview protocols will evolve during testing. The research will be iterative, in that materials’ wording and translations may change during the testing period in response to what is being learned during the interviews.

To adequately test the materials, it is necessary to distribute the cognitive interviews across respondents who represent the primary differences in experience of the target population and, correspondingly, to raise the total number of participants to obtain sufficient numbers of respondents with similar characteristics. We propose to conduct a maximum of 30 cognitive interviews with English only and Spanish-dominant respondents¹. The interviews will be distributed to at minimum include the following respondents:

1. Approximately 10 parents or guardians of children up to age 17 for interviews in English only plus another 10 for interviews in primarily Spanish, with the approximate minimum subgroup sizes for each set of 10 parents as follows:

• 2 parents or guardians of a child enrolled in grades K-12;

• 2 parents or guardians of a child ages 0 to 5;

• 3 parents or guardians with a high school completion or less education; and

• 3 parents or guardians of a child enrolled in public school and 2 in private school.

2. Approximately 5 adults, ages 18 to 65, who have a work credential for interviews in English only plus another 5 for interviews in primarily Spanish.

Revisions to the materials will be made on an ongoing basis, depending on the results of interviews conducted up to that point. Typically, we expect to conduct at least 3 interviews prior to making a change to wording or translation. Interviews will be audio-recorded. NCES staff may also observe interviews either in person or through a video-stream using WebEx video conferencing, hosted by the American Institutes for Research (AIR). The WebEx is an optional method for observing interviews and will only be used at the direction of NCES.

Methods for Testing Contact Materials

The goal of testing contact materials is to determine if respondents understand who is sending the materials, the purpose of the materials, what the study is about, and whether the respondents generally understand information included in the letters. Respondents will be given the contact materials and asked to read as if they had just received them in the mail at home.

When the respondent finishes reading, materials will be removed from the respondent’s view and they will be given an independent recall task—a series of short questions designed to explore initial reactions and the amount and quality of retained information. The goal of this exercise is to recreate the reactions to the letter as closely as possible before the respondent is asked to read the letter in great detail. These recall tasks will include questions such as:

• Do you remember who the letter is from?

• What are you expected to do with the materials sent to you?

• What is the information gathered from this study going to be used for?

After initial reactions the letters will be reviewed in more detail. Respondents in Spanish language interviews will be asked additional questions with attention to details in two areas of the letters:

1. Translation of terms, and

2. Accessibility of the language for respondents.

The protocol includes a read-aloud task in which respondents will be asked to read a full paragraph aloud, followed by specific probes on the information contained in the paragraph. The goal of these follow-up questions is to gain an understanding of the respondents’ reading abilities, explore comprehension of concepts, and identify difficult words/translations and complex syntax that may act as a barrier for respondents.

Recruitment and Paying Respondents

To assure successful recruitment of participants from all desired populations and to thank them for their time, travel, and for completing the interview, as in previous rounds of NHES cognitive interviews, each will be offered $40. Interviews will take place in the DC-Metro area (estimated 3 interviews), California (12 interviews) and in the U.S. Southwest (15 interviews) in quiet, public places, such as a library or community centers, AIR offices, or subcontractor locations. To conduct interviews in Spanish-speaking areas outside of AIR locations, AIR will subcontract with firms that offer Spanish-speaking recruitment and interviewing. Participants will be recruited by AIR and its subcontractors using multiple outreach methods and resources, such as marketing research companies, newspaper/internet ads, and contacts with schools and community organizations (e.g., libraries and summer or afterschool programs). Paper flyers, e-mails, social media, and phone calls will be used to contact potential participants (attachment 1). All materials will be distributed in English and Spanish.

Interested participants will be screened in the language that the participant is most comfortable in, either English or Spanish (see attachment 2), to ensure that they meet the participation criteria outline above, at which time the cognitive interview objectives, participation requirements, and procedures will also be explained to them. Each screening interview is estimated to take on average approximately 4 minutes.

Assurance of Confidentiality

Participation is voluntary and respondents will read a confidentiality statement and sign a consent form before interviews are conducted (see attachment 1). No personally identifiable information will be maintained after the cognitive interview analyses are completed. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the cognitive interviews to last approximately one hour and screening potential participants to last about 4 minutes. We anticipate that it will take about 15 screening interviews to yield one eligible participant, requiring about 450 screenings to yield 30 participants.

Table 1. Estimated response burden for PFI, ECPP, ATES cognitive interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	450	450	.07	32
Cognitive Interviews	30	30	1.0	30
Total	450	480	-	62

Project Schedule

Recruitment will begin as soon as OMB approval is received. Interviewing is expected to be completed before the end of May 2015. After the interviews are completed, respondent letters tested during the cognitive interviews will be revised and included in the NHES:2016 full scale package to be submitted to OMB on June 16^th, 2015.

Cost to the Federal Government

Total cost to the federal government for this cognitive laboratory study is approximately $35,000.

1 For this study, Spanish-dominant respondents are defined as respondents who indicate in the recruitment screener that they primarily speak Spanish at home.