Volume 1

2022 National Household Education Survey (NHES) Web Usability Testing

Volume I

OMB# 1850-0803 v.265

National Center for Education Statistics (NCES)

April 2020

Justification

The National Household Education Survey (NHES) is conducted by the National Center for Education Statistics (NCES) and provides descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement.

Beginning in 1991, NHES was administered approximately every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates across RDD surveys, NCES conducted a series of field tests to determine whether using self-administered mailed questionnaires would improve NHES response rates. NCES conducted the first NHES full-scale mail-out administration in 2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. NHES, which is administered in English and in Spanish, uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member is sampled for the topical survey.

In NHES:2016, a web response mode was offered in addition to the paper questionnaire. In 2017, an NHES web test was conducted. The web administration is (from the respondent’s perspective) a one-stage administration where screener questions and instantaneous online sampling lead seamlessly into the appropriate topical instrument. In NHES:2019, ECPP and PFI topical surveys were conducted using web and paper instruments. In NHES:2022, modified ECPP and PFI topical surveys will again be fielded via web and paper instruments.

The ECPP (previously conducted in 1991, 1995, 2001, 2005, 2012, and 2016, and 2019) surveys families of children ages 6 or younger who are not yet enrolled in kindergarten and provides estimates of children’s participation in three types of care arrangements: care from a relative, care from a non-relative in a private home, and care in center-based daycare or preschool programs (including Head Start and Early Head Start). Additional topics addressed in ECPP interviews have included family learning activities; out-of-pocket expenses for non-parental care; continuity of care; factors related to parental selection of care; parents’ perceptions of care quality; child health and disability; and child, parent, and household characteristics.

The PFI (previously conducted in 1996, 2003, 2007, 2012, 2016, and 2019) surveys families of children and youth enrolled in kindergarten through 12^th grade or homeschooled for these grades, with an age limit of 20 years, and addresses specific ways that families are involved in their children’s school; school practices to involve and support families; involvement with children’s homework; and involvement in education activities outside of school. Parents of homeschoolers are asked about their reasons for choosing homeschooling and resources they used in homeschooling. The 2019 PFI included new questions asking about virtual, cyber, or online coursetaking. Information about child, parent, and household characteristics is also collected.

Usability testing of the NHES:2019 web instrument uncovered some user experience issues in both the ECPP and PFI topical surveys that could not be resolved prior to that fielding due to timing and other considerations. Proposed redesigns to solve some of those issues using web-only capabilities such as complex skip patterns and auto-filled information will be tested in this study to determine if they should be included in the NHES:2022 web instruments.

NHES:2022 Web Usability & Cognitive Testing

This request is to conduct usability and cognitive testing of portions of the NHES:2022 that were identified as having user experience issues during the NHES:2019 usability testing, the CATI interviewer debriefing and/or with an expert review of the instrument conducted by Jolene Smyth¹. If the redesigned portions work well during testing, then those revisions will be considered for incorporation into the NHES:2022 web-based data collection instrument. Specific goals include testing of:

• A filter question for the ECPP questionnaire, which routes respondents to questions about the particular type(s) of care arrangement(s) specific to the sampled child;

• Automated fills to distinguish between the two parents or guardians in the Parent 1 and Parent 2 sections;

• A new disability question series for young children which aims to reflect the fact that young children are often diagnosed with “delays” rather than with specific medical conditions;

• Redesigned branching when multiple questions are on the same screen;

• Using open-text fields instead of dropdowns;

• Redesigning field labels and their location;

• Using revised questions to determine the number of grades the child was homeschooled;

• Removal of screen headers and slight changes to question text to give context without the screen headers; and

• A modified approach to ask for the child’s name.

This usability testing of the NHES:2020 web instrument will be conducted on a subset of the English language questions. The final NHES:2022 web instrument will be programmed using the Census Bureau’s standard Centurion system. That system, however, has a long development lead time and limited flexibility for the quick changes that are needed during cognitive interviewing. For that reason, the new wording and layouts will be developed using a survey software platform called Qualtrics. Qualtrics allows for faster survey development and revision by nonprogrammers. We are attaching the Qualtrics specifications as Attachment 4 which details the questions that will be used in this testing.

During test sessions, we will keep track of spontaneous comments about the questions or form features of the questionnaire. We will also note any response errors (i.e., either missing data or incorrect responses) based on observation and user verbalizations. We will gather satisfaction data from a post-survey questionnaire used in usability sessions at the Census Bureau. We will debrief each participant session, focusing on the redesigned screens and new question wordings.

The survey questions and screens will be evaluated in terms of respondent’s effectiveness and efficiency in survey completion, and satisfaction with the experience of survey completion. The primary deliverable from this study will be a report about the user results based on the updated screens and questions.

Design

After the Qualtrics instrument is complete, one round of user testing will be conducted on laptops. Depending on whether we are still practicing social distancing in June we will either conduct these user sessions over the phone using Skype for Business to see the participants’ computer screen or in person. Skype for Business is an approved Census Bureau video conferencing system. If the social distancing mandate has ended, we will conduct in-person user sessions.

During usability testing, each participant will be asked to complete one of the topical surveys (the PFI or the ECPP, as determined by the age and schooling or childcare arrangements of the respondents’ children). Participants will be asked to think aloud (verbalizing what they are thinking) as they complete the survey. Think aloud data helps identify usability problems and the causes of the problems.

Interviewers will ask probing questions as needed. In addition, pending the availability of equipment, and whether any of the sessions will be in person, the eye-tracking technique will be employed to record the participant’s visual scan and gaze pattern to help evaluate the design of the instrument. (Please note: the eye-tracking component can only occur if sessions are in person). After the completion of the survey, participants will be debriefed about their experience with the instrument. Each session is expected to last sixty minutes. See Attachment 3.

We will recruit a total of no more than 20 English-speaking participants; 10 will complete the ECPP while 10 will complete the PFI.

This study will be conducted on laptop or desktop computers and not on smartphones. Centurion does not render the questions in the same way on mobile as Qualtrics does. Consequently, it does not make sense to test on mobile devices until the screens are developed in Centurion. If the sessions are remote, the participant will use their personal computer. If the sessions are in person, the participant will use the Census Bureau provided laptop.

Each participant will complete in the Qualtrics interface an abbreviated NHES survey for their assigned topical component (i.e., either ECPP or PFI).

The following data collection methods will be used to collect participants’ performance data:

• Think-aloud protocol with minimal probing such as “Keep Talking;” “What are you thinking?” and acknowledgement tokens (linguists refer to this as backchannels) such as “Um-hum?”;

• Real-time observation by the researcher;

• Targeted probes when necessary (e.g., “What were you thinking when you answered that question?”);

• Satisfaction questionnaire;

• Retrospective debriefing;

• Audio and video recording; and

• Eye-tracking recording for in-person sessions with respondents

The in-person interviews will be audio and video-recorded. The sessions over the phone with Skype for Business will be recorded using Camtasia, a recording software on the test administrators’ laptop. The Camtasia software is installed on the test administrators’ laptop and the test participant does not interact with the software at all. However, the test participant must have Skype for Business. Any participant who does not have Skype for Business on their computer will be required to download it prior to the session. Using Skype for Business is a requirement for participating in this study. Analysis of the data will include qualitative analysis of behavioral observations, spontaneous verbalizations, and answers to debriefing questions in order to identify problems. In-person sessions will collect eye-tracking data. We will analyze recorded gaze patterns to investigate whether participants attended to or ignored important parts of the screens. Finally, we will compute the overall satisfaction ratings.

Recruiting, Interview Protocol, Instrument Specifications and Paying Respondents

To assure that we can recruit participants from all desired populations and to thank them for completing the interview, each respondent will be offered $40 for participation in a sixty-minute interview.

For the PFI participants, we will attempt to recruit households with at least one homeschooled child in grades K through 12. We will also attempt to recruit this type of households with two parents or guardians.

For the ECPP participants, we will attempt to recruit households with at least one child who is:

• not yet enrolled in Kindergarten and in a single non-parental care arrangement or

• not yet enrolled in Kindergarten and in more than one type of non-parental child care arrangement (e.g., relative, non-relative, or center based child care arrangements);

• AND (one of the following three conditions)

• has a diagnosed developmental delay (e.g., speech/language impairment, autism or on the autistic spectrum, attention disorder (ADD, ADHD), at risk for development delay) or

• does not have a diagnosed delay but parents are “concerned” that their child may have some type of developmental delay or

• does not have a diagnosed delay and parents have no such concerns about a delay.

Participants will be recruited by the U.S. Census Bureau, using multiple sources. Initially a Bureau of the Census Broadcast email will be sent. We will also post an invitation on homeschooling listservs, online neighborhood listservs, Craigslist, and social media. We will also send that invitation to personal contacts. When social distancing allows and COVID-19 closures are lifted, we will post fliers in such places as local libraries, pediatricians offices, and preschools. Recruitment contact materials including information about remote sessions being conducted on Skype for Business, information about cash payment and signed vouchers for payment are included in Attachment 1. The questions used to screen respondents for participation are included in Attachment 2.

The usability interview protocols are included in Attachment 3. The NHES questions that will appear in the Qualtrics instrument are available in Attachment 4.

In-person user sessions will be carried out in either the U.S. Census Bureau usability lab in Suitland, Maryland, or another offsite location such as a community center or library in the DC/MD/VA area. The phone sessions will occur with the test administrator in his/her home and the test participant in his/her home. For phone sessions, the test participant will also need to have access to a laptop or desktop computer, the internet and be willing to download Skype for Business. Prior to all remote sessions, we will email participants the consent form along with instructions for how to download Skype for Business. Each session will be conducted one-on-one, i.e., one participant and one test administrator, with one note taker.

Assurance of Confidentiality

Participation in this usability study is voluntary. For in-person sessions respondents will read a confidentiality statement and sign a consent form before interviews are conducted. For remote sessions, respondents will be emailed the consent form and will provide oral consent over the phone and then the recording will begin and the respondent will again provide oral consent, which will be recorded. The confidentiality statement and consent form are provided in Attachment 1. No personally identifiable information will be maintained after the usability testing interview analyses are completed. Data entered into the survey will be stored in the GovCloud in a secured FEDRamp Moderate environment with a Census Authority To Operate (ATO)² to collect PII.

The interviews will be audio and video-recorded. Participants will be assigned a unique identifier (ID), which will be created solely for data file management. The participant ID will not be linked to the participant’s name. The audio/video/eye-tracking recorded files will be secured for the duration of the study – with access limited to key U.S. Census Bureau and NCES project staff – and will be destroyed after completion of the testing. Interviews may also be observed by key project staff. Participants will be informed when observers attend.

Estimate of Hour Burden

We expect each usability interview to last approximately sixty minutes. We expect the email with instructions to download Skype for Business to take approximately 15 minutes. Screening potential participants will require 10 minutes per screening phone call. We anticipate needing to conduct 80 screening interviews to yield 20 participants for sessions. The burden on participants across both rounds of testing is a maximum total of 39 hours. If in-person interviews are conducted, the estimated burden will exclude 5 hours for downloading Skype for Business and will be about 34 hours.

Estimated response burden for 2022 NHES Web Cognitive/Usability Testing

Respondents	Number of Respondents		Number of Responses	Burden Hours per Respondent	Total Burden Hours (in hours)
Recruitment Screener	80		80	0.17	14
Email and Skype for Business download	20	*	20	.25	5
Cognitive / Usability Interview	20	*	20	1	20
Total	80		120		39

*A subset of all recruited; does not contribute to the total number of respondents

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

Recruitment will begin upon OMB approval. Interviewing is expected to be completed within 2 months of OMB approval. Please see the estimated project schedule below, built around an estimated approval date of May 6, 2020.

Estimated Project Schedule for 2022 NHES Usability Testing

Activity	Start date	End date
OMB approval	5/6/2020	5/6/2020
Respondent Recruitment (3 weeks; immediately upon approval)	5/6/2020	6/1/2020
Usability Testing (4 weeks)	6/1/2020	6/26/2020
Draft and deliver findings and recommendations	6/26/2020	7/10/2020
Interim briefing	7/10/2020	7/10/2020
NCES and ADDP review findings and recommendations	7/13/2020	7/20/2020
Prepare final report (6 weeks)	7/20/2020	8/31/2020

Cost to the Federal Government

The cost to the federal government for this usability testing laboratory study is approximately $100,000.

1 Jolene Smyth is a professor of Sociology at University of Nebraska. She is an author of numerous studies on survey design and is a nationally recognized expert on survey design.

2 This authorization means that the environment where the survey data will be stored has passed Census Bureau security standards.