Justification

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

This package requests permission to conduct 75 cognitive interviews of both the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) paper questionnaires, in order to make enhancements to these instruments in response to problems identified with current items (based on the 2016 and 2019 administrations) and in response to a series of expert reviews--from a survey design expert, from experts on disabilities within ED, and from a committee of Spanish translation experts. The changes will inform the paper questionnaires used for NHES:2022 as well as the web instruments, which will be programmed beginning in fall 2020.

Background and Study Rationale

The National Household Education Survey (NHES) is conducted by the National Center for Education Statistics (NCES) and provides descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. The NHES is conducted in both English and Spanish.

Beginning in 1991, NHES was administered approximately every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates across RDD surveys, NCES conducted a series of field tests that found that using self-administered mailed questionnaires improved NHES response rates. NCES conducted the first NHES full-scale mail-out administration in 2012, which included the ECPP and PFI questionnaires. In 2016 and 2017, NHES experimented with mailing invitations to sample members to complete the surveys via the internet (web surveys). In NHES:2019, redesigned ECPP and PFI topical questionnaires were fielded in both web and paper modes. The next administration of the NHES will be in 2022 and will again field the ECPP and PFI questionnaires in both web and paper modes.

The ECPP, previously conducted in 1991, 1995, 2001, 2005, 2012, 2016, and 2019, surveys families of children ages 6 or younger who are not yet enrolled in kindergarten. This survey provides estimates of children’s participation in care by relatives and non-relatives in private homes and in center-based daycare or preschool programs (including Head Start and Early Head Start). Additional topics addressed in ECPP interviews have included family learning activities; out-of-pocket expenses for non-parental care; continuity of care; factors related to parental selection of care; parents’ perceptions of care quality; child health and disability; and child, parent, and household characteristics. A few changes are planned for the survey instrument between the 2019 and 2022 administrations. These changes include new and revised items about the child’s disabilities, previously in the “Child’s Health” section and now being tested in the “Your Child’s Early Learning” section, based on recommendations from the Office of Special Education Programs (OSEP) and the National Center for Special Education Research (NCSER). Also planned are some revisions to survey formatting and navigation that were recommended by a questionnaire design expert, Dr. Jolene Smyth from the University of Nebraska-Lincoln. The purpose of these revisions is to reduce the design differences between the web instrument and the paper instrument and to make design elements consistent throughout the paper questionnaires. Additional revisions to items on cell phone access, the number of meals the family eats together, the child living at multiple addresses, and about the child care giver caring for the child when the child is sick have been slightly revised to provide respondents with greater clarity.

The PFI, previously conducted in 1996, 2003, 2007, 2012, 2016 and 2019, surveys families of children and youth enrolled in kindergarten through 12th grade or homeschooled for these grades, with an age limit of 20 years, and addresses specific ways that families are involved in their children’s school; school practices to involve and support families; involvement with children’s homework; and involvement in education activities outside of school. Parents of homeschoolers are asked about their reasons for choosing homeschooling and resources they use in homeschooling. In 2019, the PFI began collecting data about full-time virtual school students. Information about child, parent, and household characteristics is also collected. A few changes are planned for the survey instrument between the 2019 and 2022 administrations. These changes include new and revised items about grades for which a child is homeschooled, number of hours enrolled in school, number of meals the family eats together, child living at multiple addresses, and internet access. Also planned are some small revisions to the Child’s Health section to update terminology, as well as revisions to formatting and navigation that Dr. Smyth recommended in order to reduce the design differences between the web instrument and the paper instrument, to increase navigability through the paper instrument, and to make design elements consistent throughout the paper questionnaires. Notably, Dr. Smyth’s recommendations included a revised format for parents to select the school in which their child is enrolled.

Along with these changes, the Spanish translation went through a review between the 2019 and upcoming 2022 administrations. Multiple iterations of the questionnaires and new translations for each administration have caused the surveys to have a few inconsistent item translations over the years. Additionally, survey staff were concerned that existing translations did not adequately account for regional differences in the use of the Spanish language. Based on a review of best practices in questionnaire translation methodologies (Harkness, 2005; Pan & de la Puente, 2005), the NHES team conducted a multi-step translation process involving a team of translators, expert reviewers, and an adjudicator to determine the best translations of the items that had been flagged as either inconsistently translated or as having been problematic based on previous cognitive or usability testing results, expert review, or anomalous data editing rates from 2019. The next steps in this plan are to cognitively test translated instruments that combine old translations with improved translations for identified items.

Some of the Spanish cognitive interviews will include concept elicitation to gauge information about how to change the translation of a concept or of a question item. As opposed to typical probing in cognitive interviews, concept elicitation will ask participants about specific words that have been identified as confusing in the past and seek suggestions for alternative wordings for the item or the sentence, if applicable. The primary concept that will be tested is homeschooling, which is not a common practice in Spanish speaking countries.

Recruitment and Data Collection

Participants will be recruited by AIR using multiple sources, including Craigslist ads, social media, homeschool co-ops and listservs, relationships AIR has with Hispanic and Spanish-speaking organizations and communities, and through personal and professional contacts. An example recruitment advertisement and e-mail are included in Attachment 1. With interviews being held virtually, recruitment will be done across several different locations on Craigslist sites to ensure a diverse recruitment pool. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. Interested individuals will be asked to complete a screening questionnaire over the phone (Attachment 2). All 75 interviews will take place remotely via video conference (using the web conferencing platform GoToMeeting).

NHES:2022 English and Spanish Cognitive Interviews

Cognitive testing has been used for other NHES surveys in past years. The objective of the cognitive interviews in 2020 is to identify and correct problems of ambiguity or misunderstanding in selected question wording and in navigation of the questionnaires. The cognitive interviews should result in a set of questionnaires that are easier to understand and follow, and therefore less burdensome for respondents, while also yielding more accurate information. The primary deliverable from this study will be the revised questionnaires. A full cognitive testing report will also be delivered.

The interviews will focus on:

identifying cognitive issues (e.g., comprehension, retrieval, judgement, mapping to response options) with the new, revised, or existing Spanish translation of questions in the ECPP and PFI.
conducting concept elicitation in Spanish for the term ‘homeschooling’.
identifying cognitive issues (e.g., comprehension, retrieval, judgement, mapping to response options) with the revised PFI questions in English.
- Items for particular focus include the homeschool items, eating meal together item, child living at multiple addresses item, revised instructions about reporting about parents in the household, child’s school item, and items on internet access.
identifying any cognitive issues (e.g., comprehension, retrieval, judgement, mapping to response options) with the ECPP questions in English.
- Items for particular focus include the revised Child’s Health section, eating meal together item, child living at multiple addresses item, revised instructions about reporting about parents in the household, and items on internet access.
identifying any navigational and cognitive issues with the revised formatting and instructions in the ECPP and PFI, which includes allowing headings to appear in the middle of a column instead of always starting on a new page, spacing the not applicable response options away from scaled response options so that it is clear that the not applicable response is not a part of the scale, making sure all instructions start with a question number and end with a question to maintain the respondent’s attention to instructions, using all caps instead of underlining, rewording open-ended response labels (e.g., “[blank] books” instead of “[blank] number of books”, and adding the name of the child to the cover of the questionnaire.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to read the question out loud and “think aloud” as he or she answers survey questions, and/or to answer a series of questions about the items they just answered. Techniques include asking probing questions to clarify points that are not evident from the think-aloud comments and responding to scenarios. In these interviews, probes will be used to:

verify respondents’ interpretation of the question (e.g., asking for specific examples of reasons a child may live at another address);
check respondents’ understanding of the meaning of specific terms or phrases used in the questions (e.g., asking the parent what an IFSP or IEP means to them); and
identify experiences or concepts that the respondent did not think were covered by the questions but that we consider relevant (e.g., services their child may receive for their condition).

Some of the Spanish cognitive interviews will include concept elicitation. As opposed to typical probing in cognitive interviews, concept elicitation will ask participants about specific words that have been identified as confusing in the past, and will seek suggestions about how to change the translation of the item or the sentence, if applicable.

Each interview is expected to last about 1 hour and will be conducted by trained cognitive interviewers. Cognitive interviews conducted in Spanish will be conducted by bilingual Spanish-speaking interviewers. This submission includes the recruitment screener (Attachment 2), interview protocols that will be used to conduct the interviews (Attachment 3), and the questionnaires to be tested (Attachment 4). The research will be iterative, in that question wording and format design may change during the testing period in response to problems identified during the interviews. Thus, it is expected that the instruments and interview protocols will evolve during testing.

The interviews will be conducted in three rounds, with revisions to the items being made between the rounds based on the results of interviews conducted up to that point. Interviews will be held with parents age 18 or older who live in a household with children from birth to age 17. All interviews will be 60 minutes in length.

For the PFI interviews in both English and Spanish we will recruit parents of children enrolled in kindergarten through 12th grade and parents of children who are homeschooled and have been homeschooled prior to COVID-19 for grades equivalent to kindergarten through 12th grade. For Spanish interviews, we do not expect to find many Spanish-speaking parents of homeschoolers, but we will recruit them if they do contact us. We expect to conduct concept elicitation about homeschooling with Spanish speakers who do not homeschool.

For the ECPP interviews in both English and Spanish we will recruit parents with at least one child who is younger than 6 years old and not yet enrolled in kindergarten and:

is in a center based child care arrangement and/or
has one of the following three conditions to adequately test the child’s health section:
- has a diagnosed developmental delay (e.g., speech/language impairment, autism or on the autistic spectrum, attention disorder (ADD, ADHD), at risk for development delay) or
- does not have a diagnosed delay but parents are “concerned” that their child may have some type of developmental delay or
- does not have a diagnosed delay and parents have no such concerns about a delay.

In addition to the criteria listed above, we will ensure a mix of different participants who vary along the characteristics below:

Households with two parents or guardians.
Parent’s education levels
Race and ethnicity
Child’s care type
Child’s grade level (for PFI interviews)
Urbanicity

To adequately test the questionnaires within schedule, it is necessary to distribute the content across the 75 cognitive interviews among participants who represent the primary experiences of the target populations for the items as well as the predominant Spanish language backgrounds encountered within the U.S. population. Across the ECPP and PFI surveys, we propose to conduct 33 interviews in English and 42 interviews in Spanish. More interviews will be conducted in Spanish than English because the main priorities of this testing are to test if there are any cognitive issues on the new, revised and existing Spanish translations, to conduct concept elicitation on homeschooling in Spanish, and in order to ensure the items are tested on an appropriately wide range of Spanish dialects that are predominate in the U.S.

Table 1 provides the distribution of the English and Spanish interviews by round, topical questionnaire, and the key recruitment characteristics. Detail about the goals and design of each of the three rounds of testing follow the table.

Table 1: Overall distribution of cognitive interview participants by round, topical questionnaire, key recruitment characteristics, and language

Round and country of origin	Spanish				English					Total
	PFI	ECPP (child younger than 6)		Total Spanish	PFI		ECPP (child younger than 6)		Total English
	Parent of a child enrolled in K-12 or homeschooled¹	Parent of child in center-based care or is not concerned about a developmental delay	Parent of child diagnosed with development-al delay or concerned about development	Total Spanish	Parent of a child enrolled in K-12	Parent of a homescho-oled child	Parent of child in center-based care or is not concerned about a developmental delay	Parent of child diagnosed with developmen- tal delay or concerned about development	Total English
Round 1	14	3	4	21	3	4	3	4	14	35
Mexico	7	2	2	11	†	†	†	†	†
Puerto Rico	4	1	1	6	†	†	†	†	†
Central/South America	3	0	1	4	†	†	†	†	†
Round 2	8	3	4	15	3	3	3	3	12	27
Mexico	4	2	2	8	†	†	†	†	†
Puerto Rico	3	1	1	5	†	†	†	†	†
Central/South America	1	0	1	2	†	†	†	†	†
Round 3	3	1	2	6	3	0	2	2	7	13
Mexico	2	1	1	4	†	†	†	†	†
Puerto Rico	1	0	1	2	†	†	†	†	†
Central/South America	0	0	0	0	†	†	†	†	†
Total	25	7	10	42	9	7	8	9	33	75
† Not applicable. ¹In the Spanish interviews we do not expect to find many Spanish-speaking homeschoolers but will recruit them if they do contact us.

The following sections outline each round of interviewing and the goals of each round.

Round 1 Design

In the first round of interviewing there will be 35 interviews. Fourteen of the interviews will be in English and 21 will be in Spanish.

The primary focus of this round is to conduct concept elicitation on the term ‘homeschooling’ for the Spanish PFI interviews. Homeschooling has posed a challenge to measure among Spanish-speakers because homeschooling is not a common practice in Spanish-speaking countries. We plan to conduct 14 PFI cognitive interviews that include the concept elicitation conversation in Spanish during round 1. During these Spanish PFI interviews the participant will first discuss homeschooling with the interviewer. After completion of the concept elicitation conversation the participant will complete as much of the PFI questionnaire as possible in the remaining one hour of allocated time. This will allow us to test cognitive comprehension of the new and revised translations of the eating meal together item, child living at multiple addresses item, revised instructions about reporting about parents in the household, child’s school item, and items on internet access.

The remaining round 1 Spanish interviews will be conducted on the ECPP questionnaire (7 interviews). These interview participants will complete the entire ECPP questionnaire to identify any cognitive issues with the new, revised, or existing Spanish translations. This will include the new and revised items about disability, eating meals together, the child living at multiple addresses, revised instructions about reporting about parents in the household, and items on internet access.

Finally, 14 interviews will be conducted in English in round 1. Seven will test the PFI questionnaire and seven will test the ECPP questionnaire. Since there are minimal changes to the PFI questionnaire, the goal of this round is to primarily interview parents of children who are homeschooled prior to COVID-19 to ensure the new and revised homeschool items are clear and to interview parents of enrolled students to ensure that the school identification item is clear.

The seven English ECPP interviews in round 1 will include testing the new and revised items in the Child’s Health section, eating meal together item, child living at multiple addresses item, revised instructions about reporting about parents in the household, and items on internet access.

All interviews in the first round of testing will primarily focus on testing new and revised language, but interviewers will be instructed to probe the participant if any new formatting recommended by Dr. Smyth in both the PFI and ECPP causes confusion by the participant.

Round 2 Design

In the second round there will be a total of 27 interviews (12 English interviews and 15 Spanish interviews). For both the ECPP and PFI interviews, the second round will test revisions that resulted from the first round, continue testing the new and revised items and formatting from the first round to ensure no new cognitive issues occur, and for the Spanish PFI interviews will cognitively test any new language that was added as a result of the homeschool concept elicitation in the first round. Unlike the first round of testing, participants in the Spanish PFI interviews in round 2 will complete the entire PFI questionnaire; this round will ensure we have tested all of the PFI questionnaire in Spanish.

Round 3 Design

In the third and final round there will be a total of 13 interviews (7 English interviews and 6 Spanish interviews). For both the ECPP and PFI, the final round will focus primarily on any changes from the second round (which we expect to be minimal), will ensure that no new cognitive issues occur, and will continue to ensure that the revised formatting is clear to the participant. In this final round we are currently not planning to recruit parents of homeschooled children. If anything in round 2 indicates that we need to further test the homeschool items, we will recruit some parents of homeschooled students instead of enrolled students for round 3.

Cognitive Interview Mode

The interviews will be conducted remotely via video conference (using the web conferencing platform GoToMeeting). If a respondent would prefer that the interview be conducted over the phone, that will be accommodated, but video conference will be encouraged because nonverbal cues are important for conducting cognitive interviews. Participants will be able to join GoToMeeting either by using the GoToMeeting Web App through their browser or can download GoToMeeting to their computer. Instruction will be provided to participants on how to access GoToMeeting prior to the interview. Interviews will be audio-recorded. Interviews will be conducted primarily during regular work hours (9 am–6 pm EDT) but will be scheduled after work hours, as needed.

Due to COVID-19 and social distancing measures, we intend to conduct these interviews virtually. However, we acknowledge that as the COVID-19 situation is changing rapidly, there may be opportunities to conduct interviews in person in the summer. If the following public health criteria are met during data collection, we may begin to conduct interviews in-person:

AIR offices have opened back up to all staff
AIR offices will allow for visitors to enter offices
Public spaces are open to the public, such as libraries
A return to some normalcy with other public places, such as restaurants and stores open with normal operations – this would ensure the public may feel comfortable attending an in-person interview
Public transportation is available
State and local governments for sites we would potentially interview in-person (Metro DC, Chicago, IL, Southern CA) have lifted requirements on safe social distancing (if applicable) or bans on leaving a person’s home
Federal government/CDC restrictions on gatherings of more than 10 people and on need to be at least 6 feet apart from others lifted
Ability to ensure the safety of both the participant and the interviewer. We would plan to abide by the CDC recommendations of sanitizing the space and materials before and between each interview

We acknowledge the perceived burden and potential risk of attending a cognitive interview in-person during the current climate under the COVID-19 pandemic. If these criteria are met before the end of data collection, we will plan to send in-person recruitment materials to OMB for approval before beginning recruitment for in-person interviews.

Consultations Outside the Agency

AIR will be administering these cognitive interviews on behalf of NCES. AIR will recruit participants and conduct all the English and most of the Spanish cognitive interviews. Research Support Services, Inc. will conduct some cognitive interviews in Spanish. AIR will revise the questionnaires and write the final cognitive interview report.

Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

Paying Respondents

To assure that participants agree to take part in the interviews and to thank them for their time, each respondent will be offered a $40 gift card along with a thank-you note that will be sent via mail within 10 business days of completion of the interview.

Assurance of Confidentiality

Participation is voluntary, and respondents will read a confidentiality statement prior to completing the survey. In addition, a consent form that explains the purpose and duration of the interview will be sent via e-mail to participants, to be signed and returned prior to their interview. The confidentiality statement and consent form are provided in Attachment 1 in both English and Spanish. No personally identifiable information will be maintained after the cognitive interview analyses are completed.

The interviews will be audio-recorded. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way. Data recordings will be stored on AIR’s secure data servers and will be destroyed after completion of the testing.

Interviews may also be observed by key project staff by listening and watching the GoToMeeting session silently. The project staff will not be visible to the participant, but the interviewer will inform the participant that someone is observing the interview. At the end of the interview, the interviewer will allow for the observer to ask the participant questions if they have any.

Estimate of Hourly Burden

We expect the interviews to each be approximately one hour in length. Screening potential participants will require 3 minutes per screening. We anticipate 12 screening interviews will be required per eligible participant (thus an estimated 900 screenings to yield 75 participants). This will result in an estimated total of 120 hours of respondent burden for this study.

Table 8: Estimated response burden for NHES:2022 cognitive interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	900	900	0.05	45
Cognitive Interviews	75	75	1.0	75
Total	900	975	-	120

Estimate of Cost Burden

There is no direct cost to respondents.

Cost to the Federal Government

The cost to the federal government for this study is approximately $135,537.

Project Schedule

The project schedule calls for recruitment to begin June 1, 2020, as soon as OMB approval is received. Interviewing is expected to be completed within 4 months of OMB approval. After the interviews are completed, survey items will be finalized for the NHES:2022 administration. Table 9 provides the NHES cognitive interview activities schedule.

Table 9: Schedule for NHES:2022 cognitive interviews

Activity	Start Date	End Date
Recruitment and scheduling of participants	6/1/2020	8/14/2020
Cognitive Interviews – Round 1	6/8/2020	6/26/2020
Cognitive Interviews – Round 2	7/6/2020	7/24/2020
Cognitive Interviews – Round 3	8/3/2020	8/14/2020
Finalize questionnaires	8/17/2020	8/21/2020
Final usability testing report	8/14/2020	9/30/2020

References

Harkness, J. (2005). SHARE Translation Procedures and Translation Assessment. En A. Börsch-Supan, & H. Jürges (Edits.), The Surey of Health, Aging, and Retirement in Europe - Mehodology (págs. 24-27). Manheim, Germany: Manheim Research Institute for the Economics of Aging (MEA).

Levin, K., Willis, G. B., Forsyth, B. H., Norberg, A., Stapleton Kudela, M., Stark, D., & Thompson, F. E. (2009). Using Cognitive Interviews to Evaluate the Spanish-Language Translation of a Dietary Questionnaire. Survey Research Mehods, 3(1), 13-25.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	McQuiggan, Meghan
File Modified	0000-00-00
File Created	2021-01-14