Volume 1 - NHES 2019 Types of Schooling Cognitive Interviews - Revision 2

Volume I

National Household Education Surveys Program (NHES)

2019 Types of Schooling Cognitive Interviews

– Revision 2

OMB# 1850-0803 v.216

(revised v.212)

June 2017

revised November 2017

Background

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. The same two surveys, along with the Adult Training and Education Survey (ATES), were fielded in NHES:2016. In 2019, the NHES will field the PFI and ECPP surveys along with the second administration of the ATES. This will be a two-stage study. In the first stage, households will be screened to determine if they contain eligible members. If eligible members are in the household, within- household sampling will be performed. Finally, topical surveys will be administered to the selected household members.

The ECPP, previously conducted in 1991, 1995, 2001, 2005, 2012 and 2016, surveys families of children ages 6 or younger who are not yet enrolled in kindergarten and provides estimates of children’s participation in care by relatives and non-relatives in private homes and in center-based daycare or preschool programs (including Head Start and Early Head Start). Additional topics addressed in ECPP interviews have included family learning activities; out-of-pocket expenses for nonparental care; continuity of care; factors related to parental selection of care; parents’ perceptions of care quality; child health and disability; and child, parent, and household characteristics.

The PFI, previously conducted in 1996, 2003, 2007, 2012, and 2016, surveys families of children and youth enrolled in kindergarten through 12^th grade or homeschooled for these grades, with an age limit of 20 years, and addresses specific ways that families are involved in their children’s school; school practices to involve and support families; involvement with children’s homework; and involvement in education activities outside of school. Parents of homeschoolers are asked about their reasons for choosing homeschooling and resources they used in homeschooling. Information about child, parent, and household characteristics is also collected. To minimize response burden and potential respondent confusion, both enrolled and homeschool versions of the PFI questionnaire were created for self-administration.

The ATES, previously conducted in 2016, surveys adults ages 16 to 65 who are out of high school and provides new measures of adults’ educational and occupational credentials. It identifies adults who have educational certificates, including the subject field of the certificate, its perceived labor market value, and its role in preparing for occupational credentialing; and counts adults who have an occupational certification or license, including the number of such credentials, type of work they are for, their perceived labor market value, and the role of education in preparing for these occupational credentials. To get a comprehensive picture of adult education and training, the survey also includes brief sections on adult participation in work experience programs (such as apprenticeships) and college classes.

NHES:2019 Types of Schooling Cognitive Interviews

While enrollment in traditional, brick-and-mortar public and private schools is the dominant arrangement for formal education in the United States, other schooling arrangements, such as homeschooling and virtual schooling, also contribute to the education of the nation’s youth. Further, recent expansion in virtual schooling and the comingling of homeschooling and virtual schooling necessitate a renewed examination of these education models along a number of dimensions, including: motivations for opting out of brick-and-mortar schools; populations of students served; patterns of teaching, learning, curriculum, and assessments; policy issues; and outcomes for students. Currently, research about homeschooling and virtual schooling faces several challenges. For example, it is difficult to study homeschooling and virtual schooling due to limited administrative data available on these topics and due to the overlap among the schooling options.

In 2012 and 2016, the PFI survey utilized separate survey instruments for parents of students enrolled in a brick-and-mortar school and parents of homeschooled students. The NHES program observed increasing evidence of data quality issues with this structure. Analyses uncovered the fact that the screener was not able to adequately sort respondents into enrolled students and homeschooled students; often, responses to the topical surveys indicated that the child was not identified accurately as a homeschooled student or as an enrolled student. One reason for these challenges may be the growth in virtual education, wherein students are educated at home but using curricula provided by a school. In 2019, the NHES program would like to move away from reliance on the screener instrument to identify homeschooled students. Instead, the screener will identify school-aged students. Identification of the child as a homeschooled student, virtual education student, enrolled student, or combination thereof, will take place through analysis of responses to questions on a “combined” PFI survey instrument.

The request to conduct cognitive interviews to assess the extent to which draft NHES:PFI survey items for 2019 capture respondents’ perceptions regarding homeschooling, virtual schooling, and school choice phenomena, and to evaluate respondents’ ability to navigate the PFI questionnaire appropriately was approved in June 2017 with a revision approved in October 2017 (OMB #1850-0803 v. 200 & 212). The key points of interest for this study are (a) respondents’ experiences with a combined PFI that includes questions for both families that homeschool and those who enroll their children in public or private schools and (b) performance of any new items that have not been part of PFI data collection in prior administrations, particularly new items related to virtual education, homeschooling, and school choice. Due to significant challenges encountered in recruiting participants, this request is to reduce the planned number of interviews and to create subgroup targeted advertising.

Design

Each cognitive interview will explore parents’ reactions to and thoughts concerning the order and wording of new, reworded, and existing items on the proposed NHES:2019 combined PFI instrument. The interviews will be conducted using a team of approximately 15 staff from Sanametrix, Activate Research, and Child Trends experienced with qualitative interviewing techniques, open-ended probes, and semi-structured interview protocols. To the extent possible, the interviews will be conducted in-person in the metropolitan Washington, DC area in various locations (library, community center, coffee house, etc.) as is convenient for each participant. However, due to the rare nature of the population being studied, some of the interviews may need to be conducted over the phone or through the use of video-conference technology. The protocol for the cognitive interviews is included in Attachment 2, and the items to be tested using the protocol are included in Attachment 3.

An initial round of about 10 interviews roughly equally distributed across three school arrangements [(1) parents who homeschool their child(ren), (2) parents whose child(ren) attends a school located in a physical building and also use virtual education, and (3) parents whose child(ren) attend a school other than their assigned public school] will be conducted to assess if the combined PFI survey instrument is feasible in terms of respondents’ ability to navigate the instrument’s skip patterns and expanded length. The remaining interviews will focus on the new and reworded items, particularly in the area of virtual education, homeschooling, and school choice. A total of 48 cognitive interviews will be conducted. All interviews will look at instrument navigation, with 10 focusing on instrument navigation and the rest on testing new content. At least 5 interviews will be conducted with parents of homeschoolers, at least 5 with parents of students enrolled in a public or private school who utilize some virtual education, at least 2 parents of students whose school is not their assigned public school (i.e., utilizing school choice), and at least 5 interviews with parents of students enrolled in their assigned public school that does not utilize virtual education (target is 10 interviews in each group conducted in English and 2 interviews in Spanish; see Table 1). If participants consent, the interviews will be audio recorded for the purposes of summarizing the results. Survey staff from the NHES project team may also be in attendance as observers, and cognitive interview participants will be informed before the interview about observers. Any materials that are generated during the interviews will be captured and incorporated into the results as well.

Participants for cognitive interviews will be diverse with respect to race/ethnicity, educational attainment, and income. As such, a tally of race/ethnicity, education, and household income of each recruited participant will be kept ensuring roughly equal representation of these demographic characteristics across the groupings. However, parents’ race/ethnicity, educational attainment, and household income will not be used as quotas for recruitment. To determine whether the combined PFI items pose any systematic problems, recruited parents will be grouped by virtual school arrangement and their children’s main language. While not being used as quotas for recruitment, information on the child’s age/grade and sex will be collected in the screener so that some diversity in children as well as parents can be obtained.

Table 1. Grouping of Types of Schooling for Cognitive Interviews

Interview type and school arrangement	Language of parent		Total
	English	Spanish
Total	40	8	48
Interview type
Navigation	10	0	10
New content	30	8	38
School arrangement
Homeschool	10	2	12
Public/Private school and virtual school	10	2	12
Public/Private school with school choice	10	2	12
Public school without virtual school or school choice	10	2	12

NOTE: Care will be exercised to obtain a mix of race/ethnicity, education, and income groups; however, these demographic characteristics are not being used as quotas in the recruitment plan.

Recruiting and Paying Respondents

Participants will be recruited by Sanametrix, using multiple sources, including company databases, social media/Craig’s List, personal and professional contacts, and on-the-ground recruitment (handing out study flyers and describing the study to potential participants – Spanish language only). A partnership with one or more virtual school associations will be forged to help identify parents of children attending school virtually.

Additionally, we will identify DC metropolitan area schools and school districts in which online course taking occurs but, because of the short timeline of this study, for which their respective districts do not require lengthy approval processes. We will then request permission from the districts of the desired schools to contact the schools directly, and will then ask the schools to distribute a study recruitment flier to students to take home to their parents. Separately, we will also contact parent teacher associations (PTAs) and parent teacher organizations (PTOs) for local schools and asked them to distribute the flier to their membership. All recruitment materials (e-mail, social media, letters, and flyers) are included in Attachment 1.

To determine their eligibility for the study, a brief screener interview will be conducted either by telephone or by video conference with those who express an interest in participating in the cognitive interviews. If a potential participant qualifies for the study, a cognitive interview will be scheduled at a time that is convenient for him or her. Recruited participants will receive a reminder call/email/text the day before the cognitive interview. The questions used to screen respondents for participation as well as the reminder call/email/text script are included in Attachment 1. People who have participated in usability testing, cognitive studies, or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating.

To assure that we are able to recruit the difficult to recruit, select types of participants who are representative of homeschooling, virtual schooling, and school choice, who are like the parents that will take part in the 2019 NHES, and to thank them for their time and for completing the interview, during recruitment each parent will be offered a $75 incentive for participation. In-person participants will be paid in cash at the interview, while remote (phone or video conference) participants will be paid by a check mailed after the interview. Participants will be asked to acknowledge their receipt of the incentive (See Attachment 1).

Assurance of Confidentiality

The study will not retain any personally identifiable information. Prior to the start of the study, participants will be notified that their participation is voluntary. As part of the study, participants will be notified that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

For all participants, written consent will be obtained before interviews are administered. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is released. Interviews may be recorded using audio technology. The only identification included on the files will be the ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

Estimate of Hour Burden

We expect each cognitive interview to last approximately 60 minutes. Recruiting participants will require 5 minutes per potential participant. We anticipate it will require 35 recruitment attempts per eligible participant (thus an estimated 1,680 attempts to yield 48 participants). This will result in an estimated total of 188 hours of respondent burden for this study.

Table 2. Estimated response burden for 2017 NHES Types of Schooling Cognitive Interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment	1,680	1,680	0.0833	140
Cognitive Interviews	48	48	1	48
Total	1,680	1,728	-	188

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for recruitment to begin as soon as OMB approval is obtained. The interviews are expected to be completed by November 2017.

Cost to the Federal Government

The cost to the federal government for this cognitive interview study is approximately $118,724.