Vol. 1 - Justification

Volume I

2016 National Household Education Surveys Program (NHES)

Spanish Language Cognitive Interviews

Phase 2

OMB# 1850-0803 v.131

March 16, 2015

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. Such topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, attainment of non-degree credentials, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. In 2016, the NHES will again field the PFI and the ECPP, plus the Adult Training and Education Survey (ATES). This will be a two-stage mail study. In the first stage, households will be screened to determine if they contain eligible members. If eligible members are in the household, within household sampling will be performed. Finally, topical surveys will be administered to the selected household members. Out-of-high-school adults ages 16 to 65 will be eligible to receive the ATES.

In order to reduce the respondent’s cognitive and time burden and to accommodate the materials development schedule, we have divided testing of Spanish NHES survey materials into three phases. This request is to conduct the phase 2 interviews. Phase 1 cognitive interviews tested the language and translation of Spanish letters, postcards, other contact materials, screener instruments, and a few items from the ATES that have not been translated previously. Phase 2 focuses on the topical questionnaires as translated for the prior NHES in 2012, plus new and revised translations. Phase 3 will focus on contact materials for a web experiment planned for NHES:2016 (expected OMB submission in April 2015).

We will test the following questionnaires in phase 2:

• Early Childhood Program Participation (ECPP)

• Parent and Family Involvement in Education – Enrolled (PFI)

• Parent and Family Involvement in Education – Homeschool (PFI)

• Addendum to parent questionnaires based on recent (March 13, 2015) cognitive interview-based revisions

• Adult Training and Education Survey (ATES)

Note about ESRA and paperwork statement language. The Phase 2 interviews are not testing ESRA or screener language. This language is currently being tested as part of Phase 1. Any changes from 2012 will be requested as part of the NHES:2016 full survey OMB submission and are not reflected in the attached 2012 surveys in English or Spanish.

Cognitive testing has been used for other NHES surveys in past years. The objective of this round of cognitive interviews is to identify and correct problems of ambiguity or misunderstanding in the questionnaires that are particular to language translation. This should result in questionnaires that are easier to understand and therefore less burdensome for respondents.

Materials for recruiting cognitive interview participants and participation consent forms are provided in attachment 1, recruitment screener in attachment 2, and the cognitive interview protocol in attachment 3. The NHES:2016 questionnaires are provided in attachment 4.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to answer a series of questions about the materials they have just interacted with. Respondents will be asked to “think aloud” as they answer survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions, as necessary, clarifying points that are not evident from the think-aloud comments, and responding to scenarios.

Probing will focus on questions for which translations differ from the NHES:2012 administration, although all translation issues that arise will be evaluated.

The following types of probes will be used:

• to verify respondent’s interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating in);

• to verify respondent’s understanding of the meaning of specific terms or phrases used in the questions; and

• to identify experiences or concepts that the respondent did not think were covered by the questions but we consider relevant.

Interviews are expected to last about 1 hour and will be conducted by trained bilingual (Spanish/English) cognitive interviewers in Spanish. It is expected that the instruments and interview protocols will evolve during testing. The research will be iterative, in that question wording, and format or design may change during the testing period in response to what is being learned during the interviews.

To adequately test the instruments, it is necessary to distribute the cognitive interviews across respondents who represent the primary differences in experience of the target population and, correspondingly, to raise the total number of participants to obtain sufficient numbers of respondents with similar characteristics. The interviews will be distributed to, at minimum, include the following respondents:

• Approximately 60 parents or guardians of children up to age 17, with the approximate subgroup sizes as follows:

  • 20 parents or guardians of a child enrolled in public school and 12 in private school;

  • 5 parents or guardians of a child who is homeschooled;

  • 23 parents or guardians of a child ages 0 to 5 and not yet in kindergarten, 14 of whom have a care arrangement for the child;

  • 10 parents or guardians will have a high school completion or less education.

• Approximately 20 adults, ages 18 to 65, who have a work credential.

Note: The sum of the categories listed above is greater than the total because several respondents are expected to fall into multiple categories.

Revisions to the questionnaire will be made on an ongoing basis, depending on the results of interviews conducted up to that point. Typically, we expect to conduct at least 3 interviews prior to making a change to question wording. Interviews will be audio-recorded. NCES staff may also observe interviews either in person or through a video-stream using WebEx video conferencing, for interviews conducted at the American Institutes for Research (AIR) offices. The WebEx is an optional method for observing interviews and will only be used at the direction of NCES.

Recruitment and Paying Respondents

To assure successful recruitment of participants from all desired populations and to thank them for their time, travel, and for completing the interview, as in previous rounds of NHES cognitive interviews, each will be offered $40. Interviews will take place in the DC-Metro area (estimated 10 interviews), San Mateo area (estimated 20 interviews) and in the U.S. Southwest (50 interviews) in quiet, public places, such as a library or community centers, AIR offices, or subcontractor locations. To conduct interviews in Spanish-speaking areas outside of AIR locations, AIR will subcontract with firms that offer Spanish-speaking recruitment and interviewing. Participants will be recruited by AIR and its subcontractors using multiple outreach methods and resources, such as marketing research companies, newspaper/internet ads, and contacts with schools and community organizations (e.g., libraries and summer or afterschool programs). Paper flyers, e-mails, social media, and phone calls will be used to contact potential participants (attachment 1). All materials will be distributed in Spanish.

Interested participants will be screened in Spanish (see attachment 2) to ensure that they meet the participation criteria outline above, at which time the cognitive interview objectives, participation requirements, and procedures will also be explained to them. Each screening interview is estimated to take on average approximately 4 minutes.

Assurance of Confidentiality

Participation is voluntary and respondents will read a confidentiality statement and sign a consent form before interviews are conducted. The confidentiality statement and consent form are provided in English and Spanish in attachment 1. No personally identifiable information will be maintained after the cognitive interview analyses are completed. Primary interview data will be destroyed on or before December 31, 2015. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the cognitive interviews to be approximately one hour in length. Screening potential participants will require about 4 minutes per screening. We anticipate it will require 15 screening interviews per eligible participant (thus an estimated 1200 screenings to yield 80 participants). This will result in 84 hours of burden for the screener, and an estimated total 164 hours of respondent burden for this study.

Table 1. Estimated response burden cognitive interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	1200	1200	0.07	84
Cognitive Interviews	80	80	1.0	80
Total	1200	1280	-	164

Project Schedule

The project schedule calls for recruitment to begin as soon as OMB approval is received in March 2015. Interviewing is expected to be completed within 3 months of OMB approval. After the interviews are completed, NHES:2016 data collection instruments will be revised.

Cost to the Federal Government

The cost to the federal government for testing the questionnaires is approximately $130,000.