Justification

Volume I

2016 National Household Education Surveys Program (NHES)

Spanish Language Cognitive Interviews

Phase I

OMB# 1850-0803 v.128

February 9, 2015

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. Such topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, attainment of non-degree credentials, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. In 2016, the NHES will again field the PFI and the ECPP, plus the Adult Training and Education Survey (ATES). This will be a two-stage mail study. In the first stage, households will be screened to determine if they contain eligible members. If eligible members are in the household, within household sampling will be performed. Finally, topical surveys will be administered to the selected household members. Out-of-high-school adults ages 16 to 65 will be eligible to receive the ATES.

In order to reduce the respondent’s cognitive and time burden and to accommodate the materials development schedule, we have divided testing of Spanish NHES survey materials into three phases. This request is to conduct the phase 1 cognitive interviews to test and revise the language and translation of Spanish letters, postcards, other contact materials, screener instruments, and a few items from the ATES that have not been translated previously. Phase 2 will focus on the topical questionnaires, and phase 3 on contact materials for a web experiment planned for NHES:2016.

We will test only the initial version of each type of communications material because the language used in nonresponse follow-up communications is very similar to the initial letter. We will test the following:

• Initial screener mailing envelope,

• Initial screener letter,

• Screener postcard reminder,

• Screener questionnaire,

• Initial topical letter (Either ECPP, ATES, or PFI-Enrolled depending on the respondent type), and

• Select ATES items.

Cognitive testing has been used for other NHES surveys in past years. The objective of this round of cognitive interviews is to identify and correct problems of ambiguity or misunderstanding in respondent contact materials particular to language translation. This should result in a set of contact and data collection materials that are easier to understand and therefore less burdensome for respondents.

Materials for recruiting cognitive interview participants and participation consent forms are provided in attachment 1, recruitment screener in attachment 2, and the cognitive interview protocol in attachment 3. The NHES:2016 respondent contact materials to be tested (envelope, postcards, and letters) are provided in attachment 4, and the NHES:2016 screener and ATES questionnaire items to be tested in attachment 5.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to answer a series of questions about the materials they have just interacted with. Techniques include asking probing questions, as necessary, clarifying points, and responding to scenarios. Interviews are expected to last about 1 hour and will be conducted by trained bilingual (Spanish/English) cognitive interviewers in Spanish. It is expected that the instruments and interview protocols will evolve during testing. The research will be iterative, in that contact materials, question wording, and format design may change during the testing period in response to what is being learned during the interviews.

To adequately test the materials, it is necessary to distribute the cognitive interviews across respondents who represent the primary differences in experience of the target population and, correspondingly, to raise the total number of participants to obtain sufficient numbers of respondents with similar characteristics. We propose to conduct a maximum of 30 cognitive interviews with Spanish-dominant respondents¹. The interviews will be distributed to at minimum include the following respondents:

1. Approximately 20 parents or guardians of children up to age 17, with the approximate minimum subgroup sizes as follows:

• 8 parents or guardians of a child enrolled in grades K-12;

• 5 parents or guardians of a child ages 0 to 5;

• 5 parents or guardians with a high school completion or less education; and

• 10 parents or guardians of a child enrolled in public school and 5 in private school.

2. Approximately 10 adults, ages 18 to 65, who have a work credential.

Note: The sum of the categories listed above is greater than the total because several respondents are expected to fall into multiple categories.

Revisions to the questionnaire will be made on an ongoing basis, depending on the results of interviews conducted up to that point. Typically, we expect to conduct at least 3 interviews prior to making a change to question wording. Interviews will be audio-recorded. NCES staff may also observe interviews either in person or through a video-stream using WebEx video conferencing, hosted by the American Institutes for Research (AIR). The WebEx is an optional method for observing interviews and will only be used at the direction of NCES.

Methods for Testing Contact Materials

The goal of testing contact materials is to determine if respondents understand who is sending the materials, the purpose of the materials, what the study is about, and whether the respondents understand the privacy language translation included in the letters. Respondents will be given the contact materials and asked to read as if they had just received them in the mail at home.

When the respondent finishes reading, materials will be removed from the respondent’s view and they will be given an independent recall task—a series of short questions designed to explore initial reactions and the amount and quality of retained information. The goal of this exercise is to recreate the reactions to the letter as closely as possible before the respondent is asked to read the letter in great detail. These recall tasks will include questions such as:

• Do you remember who the letter is from?

• What are you expected to do with the materials sent to you?

• What is the information gathered from this study going to be used for?

After initial reactions the letters will be reviewed in more detail. Respondents will be asked additional questions with attention to details in two areas of the letters:

1. Translation of terms, and

2. Accessibility of the language for respondents.

The protocol includes a read-aloud task in which respondents will be asked to read a full paragraph aloud, followed by specific probes on the information contained in the paragraph. The goal of these follow-up questions is to gain an understanding of the respondents’ reading abilities, explore comprehension of concepts, and identify difficult words/translations and complex syntax that may act as a barrier for respondents.

Method for Testing Screener Materials and ATES Items

After respondents complete the contact materials task, the interviewer will ask the respondent to fill out the screener questionnaire, followed by the ATES items. These materials will be tested using a primarily think aloud procedure. Techniques include asking probing questions, as necessary, clarifying points that are not evident from the think-aloud comments, and responding to scenarios. The research will be iterative, in that question wording and forms design may change during the testing period in response to early findings. Probes that will be used include:

• Probes to verify respondents’ interpretation of the question;

• Probes about respondents’ understanding of the meaning of specific terms or phrases used in the questions; and

• Probes to identify words that are mistranslated or that create misunderstanding for the respondent in Spanish.

At the end of the interview, if needed, debriefing questions will be administered to revisit key concepts and help researchers understand how respondents are linking the concepts of the letters with the screener materials.

ATES items to be tested in Phase 1:

We will test two sets of questions from the adult topical questionnaire ATES in Phase 1 of the 2015 NHES Spanish Language Cognitive Interviews. The items that will be included in each set are outlined below. They will be administered as part of a “mini” questionnaire in order to provide sufficient context. The questionnaire is provided in Attachment 5.

SET 1 – CERTIFICATION AND LICENSE HOLDERS (n=5)

	Previous translation of item	Current item in English	Problematic word/phrase to be retranslated and tested
1	Question text: ¿Cuán útil ha sido su última certificación o licencia para cada una de las siguientes situaciones? a. Conseguir un empleo b. Mantener un empleo c. Mantener su trabajo vendible con empleadores o clientes d. Mejorar sus destrezas para el trabajo Response options: No fue útil Un poco útil Muy útil Muy pronto para saber	Question text: How useful has your most important certification or license been for each of the following? a. Getting a job b. Keeping a job c. Keeping you marketable to employers or clients d. Improving your work skills Response options: Not useful Somewhat useful Very useful Too soon to tell	Problematic words/phrases: Destrezas (skills) Vendible (marketable) Other changes: To ensure consistency with the most recent version of the English questionnaire, the question text will be retranslated from “How useful has your last certification or license” to “How useful has your most important certification or license been…”
2	Question text: ¿Hizo usted alguna de las siguientes cosas en preparación para obtener su última certificación o licencia? d. Tomó clases en una universidad, escuela técnica o escuela vocacional (sin título) e. Participó en capacitación en el trabajo, pasantía o internship , o un programa de aprendiz. Response options: No/Sí	Question text: Did you do any of the following to prepare for getting your most important certification or license? d. Took classes from a college, technical school, or trade school e. Participated in on-the-job training, an internship, or apprenticeship…. Response options: No/Yes	Problematic words/phrases: Capacitación (training) Pasantia (internship) Apprenticeship Vocacional Other changes: To ensure consistency with the most recent version of the English questionnaire, the question text will be retranslated from “to prepare for earning your last certification or license” to “to prepare for getting your most important certification or license” and remove the parenthetical for part d of the question.

SET 2 – PEOPLE WHO HAVE COMPLETED A WORK EXPERIENCE PROGRAM (n=5)

	Previous translation of item	Current item in English	Problematic word/phrase to be retranslated and tested
1	Question text: Algunas veces las personas se preparan para entrar a un oficio o una profesión por medio de un programa que combina experiencia de trabajo con la educación o capacitación formal. ¿Alguna vez ha COMPLETADO uno de estos tipos de programas de experiencia de trabajo, tal como pasantía o internship, prácticas para maestros, Programa de cooperativa, asistente jurídico o medico (clerkship), residencia médica, experiencia clínica, programa de aprendiz? Response options: No/Sí	Question text: People sometimes prepare to enter a trade or profession through a program that combines work experience with education or formal training. Have you ever COMPLETED this type of work experience program—for example, an internship, student teaching, co-op, practicum, clerkship, externship, residency, clinical experience, or apprenticeship? Response options: No, and I am not in one now No, but I am in one now Yes, I have completed this type of program	Problematic words/phrases: Capacitación (training) Pasantia (internship) Apprenticeship Other changes: To ensure consistency with the most recent version of the English questionnaire, the question text will be retranslated from “Have you ever COMPLETED one of these types of work experience program” to “have you ever COMPLETED this type of work experience program.”
2	Question text: Si contestó que sí, ¿describe algunas de las siguientes oraciones su último programa de experiencia de trabajo? c. Recibí clasificación de obrero especializado al completar el aprendizaje d. Recibí un número de registro de aprendiz del gobierno estatal o federal Response options: No/Sí	Question text: Do the following statements describe your last work experience program? c. I received journeyman status at the end of an apprenticeship d. I got a state or federal apprenticeship registration number Response options: No/Yes	Confusing terms: Journeyman Status Apprenticeship Other changes: To ensure consistency with the most recent version of the English questionnaire, the question text will be retranslated from “If yes, do any of the following…” to “Do any of the following…”
3	Question text: ¿Cuán útil ha sido su última [work experience program] para cada una de las siguientes situaciones? a. Conseguir un empleo b. Mantener un empleo c. Mejorar sus destrezas para el trabajo Response options: No fue útil Un poco útil Muy útil Muy pronto para saber	Question text: How useful was your last work experience program for each of the following? a. Getting a job b. Keeping a job c. Improving your work skills Response options: Not useful Somewhat useful Very useful Too soon to tell	Problematic words/phrases: Destrezas (skills) Other changes: The question stem (“How useful was your last work experience program for each of the following?”) is partially translated; we will need to substitute translation of “work experience program” for translation of “certification or license” see Q1 in section above)].

Recruitment and Paying Respondents

To assure successful recruitment of participants from all desired populations and to thank them for their time, travel, and for completing the interview, as in previous rounds of NHES cognitive interviews, each will be offered $40. Interviews will take place in the DC-Metro area (estimated 8 interviews) and in the U.S. Southwest (22 interviews) in quiet, public places, such as a library or community centers, AIR offices, or subcontractor locations. To conduct interviews in Spanish-speaking areas outside of AIR locations, AIR will subcontract with firms that offer Spanish-speaking recruitment and interviewing. Participants will be recruited by AIR and its subcontractors using multiple outreach methods and resources, such as marketing research companies, newspaper/internet ads, and contacts with schools and community organizations (e.g., libraries and summer or afterschool programs). Paper flyers, e-mails, social media, and phone calls will be used to contact potential participants (attachment 1). All materials will be distributed in Spanish.

Interested participants will be screened in Spanish (see attachment 2) to ensure that they meet the participation criteria outline above, at which time the cognitive interview objectives, participation requirements, and procedures will also be explained to them. Each screening interview is estimated to take on average approximately 4 minutes.

Assurance of Confidentiality

Participation is voluntary and respondents will read a confidentiality statement and sign a consent form before interviews are conducted. The confidentiality statement and consent form are provided in English and Spanish in attachment 1. No personally identifiable information will be maintained after the cognitive interview analyses are completed. Primary interview data will be destroyed on or before December 31, 2015. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the cognitive interviews to last approximately one hour and screening potential participants about 4 minutes. We anticipate the need to conduct about 15 screening interviews to yield one eligible participant (i.e. about 450 screenings to yield 30 participants).

Table 1. Estimated response burden for PFI, ECPP, ATES cognitive interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	450	450	.07	32
Cognitive Interviews	30	30	1.0	30
Total	450	480	-	62

Project Schedule

Recruitment will begin as soon as OMB approval is received. Interviewing is expected to be completed within 4 months of OMB approval. After the interviews are completed, respondent contact materials, screener, and questions tested during the cognitive interviews will be revised.

Cost to the Federal Government

Total cost to the federal government for this cognitive laboratory study is approximately $35,000.

1 For this study, Spanish-dominant respondents are defined as respondents who indicate in the recruitment screener that they primarily speak Spanish at home.