Justification - Volume 1 ECLS-K2023 Preschool FT Instruments Usability Testing

The Early Childhood Longitudinal Study, Kindergarten Class of 2022-23 (ECLS-K:2023) Preschool Field Test Instruments Usability Testing

Volume I

OMB #1850-0803 v.253

Submitted by

National Center for Education Statistics

U.S. Department of Education

August 2019

Attachments:

Attachment 1: Preschool field test spring instruments

Attachment 2: Recruitment screener

Attachment 3: Usability testing protocol

Attachment 4: Participant consent form

Attachment 5: Preschool field test MyECLS website screens

Attachment 6: Preschool field test initial letter

Justification

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing such as usability tests, focus groups, and cognitive interviews to improve methodologies, survey questions, and/or delivery methods.

NCES conducts the Early Childhood Longitudinal Study (ECLS) program to provide descriptive data on the educational activities of the U.S. population. The ECLS program provides national data on children's status at birth and at various points thereafter; children's transitions to nonparental care, early education programs, and school; and children's experiences and growth through the eighth grade. The Early Childhood Longitudinal Study, Kindergarten Class of 2022-23 (ECLS-K:2023) will be the fourth cohort in the series of early childhood longitudinal studies that began with the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) and continued with the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B) and the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011). The purpose of the ECLS-K:2023 will be to provide important information on children’s early learning and development, preschool early care and education experiences, transition into kindergarten, and progress through the elementary grades, building upon knowledge acquired from the previous Early Childhood Longitudinal Studies. Data from the ECLS-K:2023 will yield policy-relevant information by (1) providing current data on the cohort of students to be selected in the early 2020’s and their progress through the elementary grades, (2) providing data relevant to emerging policy related domains not measured fully in the prior ECLS studies, and (3) allowing cross-cohort comparisons of the educational experiences of children attending school before and after the advent of new federal laws and policies. The data collections will be carried out for NCES by Westat, with the Educational Testing Service (ETS) as the subcontractor developing the child assessments and Hager Sharp as the subcontractor developing the respondent communication materials.

As in its sister studies, the ECLS-K and the ECLS-K:2011, the ECLS-K:2023 is designed to provide data that can be used to analyze the relationships between a wide range of family, school, community, and individual variables and children's development, early learning, and performance in school. Information will be collected directly from the child, including the use of direct cognitive assessments with the children in reading and math, and in the later rounds, child questionnaires. Information will also be collected from the child’s parents/guardians, teachers, and school administrators. While all of these components will be included in the study, the ECLS-K:2023 will be designed such that the child will be the unit of analysis; the study will also be representative at the school level at the kindergarten year.

The ECLS-K:2023 will be a kindergarten cohort study (as such, “base year” for the study is considered the students’ kindergarten year); however, the ECLS-K:2023 will go beyond its predecessor kindergarten cohort studies by adding a preschool round of data collection. In the spring of 2022, households in selected primary sampling units (PSUs) will be geocoded and screened for the presence of a child who will be in kindergarten the following fall, in which cases web surveys (with a paper survey option) will be collected from the child’s parent or guardian. These children will then be added to the cohort of 2022-23 kindergartners. Collecting parent data starting in preschool enables the study to measure influences on children before school entry, including their home environments and access to early care and education. The preschool data collection procedures will be field tested in spring 2020. A full OMB clearance package for this field test began the clearance process in July 2019 (OMB #1850-0750 v19).

This request is to conduct in fall 2019 usability testing with parents of preschool children (i.e., children who will attend kindergarten in the next school year) to obtain feedback on the functionality of the preschool household screener and spring parent survey that will be used in the ECLS-K:2023 preschool field test, planned to begin in spring 2020. The results of the usability testing will guide adaptations to the final screener and surveys used in the field test and, combined with the results of the field test, of those that will be used in the main study preschool data collection, planned to begin in spring 2022.

Design

Prior to the 2020 preschool field test, the spring online surveys to be administered in that test (i.e., a household screener and the spring parent survey) will undergo usability testing. Attachment 1 contains the instruments that will be tested, assuming programming of all items is completed at the time of usability testing. If programming is not complete at the time of testing, a subset of these items will be included. The testing will evaluate the functionality of the online mode, especially the navigational instructions, layout of the screens, and entering and exiting the surveys. During the testing, if respondents provide feedback on the wording or content of the items, these comments will be noted and considered for revisions for the main study instruments to be fielded to the national sample in early 2022. Areas of confusion, errors, corrections to misunderstandings, and use of help text will be noted. Various browsers will be tested, as well as the use of a computer versus a mobile device to complete the surveys.

Westat will post requests for participants on Westat’s internal website, and other local parent group websites as needed, for 20 local parents of preschool age children to visit Westat’s usability lab for 90-minute sessions. Efforts will be made to recruit respondents from a variety of backgrounds, including respondents with a mix of experience and comfort using technology. Attachment 2 contains the screener that will be used to recruit respondents.

During the testing, respondents will be asked to complete the web surveys on a laptop/desktop computer or a mobile device (smartphone or tablet) while a project team member sits with the respondent, observes, and asks questions. Other observers can watch behind a one-way glass panel or see video recordings after the usability testing has been completed. Each respondent will be asked to complete the spring preschool field test instruments (i.e., the household screener and spring parent survey) as time allows.

Respondents will be randomly placed into one of two groups. The two groups will use two different cognitive interviewing techniques. The first group will utilize retrospective interviewing. The second group will utilize think-aloud interviewing. The reasons for structuring the cognitive labs in this manner are to obtain accurate survey timing data from the first group, and real-time feedback about the ease or difficulty of use (i.e., how easy or difficult it is to take the survey) from the second group.

The first group will complete the screener and surveys without interruption from the project team member, though respondents may provide verbal feedback if they choose to while completing the surveys. Half of this group will complete the screener and surveys on a mobile device browser and the other half will complete them on the computer browser, with different operating systems and browsers represented. Timings to complete each instrument will be collected from these users. At the end of the testing session, the respondents will be shown screenshots of pre-selected items from the surveys that study staff believe may prove challenging to answer. The usability testing participants will be asked follow-up questions about these items to get more information about their experience completing these items. These questions are listed in Attachment 3.

The second group will also complete the screener and surveys but will be asked to think out loud and answer follow-up questions as they progress. That is, after pre-selected items are completed by the respondent, the observer will stop the respondent and ask questions about the item, probing to understand the respondent’s thoughts and experiences with the item. The respondent will then be instructed to continue until the next pre-selected item is reached. The pre-selected items will be the same as used in the first group, but this group will be asked to answer questions about the items as they are completing the survey, rather than waiting until the survey is completed. As in the first group, this group will also be divided such that some respondents take the surveys on computers and others on mobile devices.

In addition, for both groups, a variety of techniques will be used while the respondent completes the web survey to collect the needed usability testing data, including observations of how the survey is being completed, recording keystrokes, and debriefing after the survey. Attachment 3 contains the protocol that will be used as part of the usability testing.

In the event that programming for the full survey is not complete at the time of usability testing, an additional group of respondents may be recruited to complete the survey after the programming is completed so that accurate timings can be collected.

Recruiting and Paying Respondents

Westat will recruit for and host the usability testing sessions at Westat’s facility in Rockville, MD. Project staff will contact by telephone people who respond to the advertisement posted on a company-wide internal website, and other local parent group websites as needed, and administer to them the recruitment screener (attachment 2) to identify those who meet the eligibility criteria, namely that they are a parent or legal guardian of at least one preschool child who will attend kindergarten or a kindergarten grade equivalent in the next school year. The recruiters will also gather basic demographic information from parent participants.

To ensure the recruitment of participants who are representative of typical ECLS sample members and to thank them for their time and for participating in the focus group session, during recruitment each parent will be offered a $50 cash incentive for participation.

Assurance of Confidentiality

The statement below will be presented to all participants, including on the consent form (Attachment 4) that will be collected from each participant before the usability testing. Participants will also be informed that they can leave the testing at any time.

The National Center for Education Statistics (NCES) is authorized to conduct the Early Childhood Longitudinal Study (ECLS) by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). The data are being collected for NCES by Westat, a U.S.-based research organization. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

All Westat staff working on the ECLS-K:2023 usability tests have already signed an Affidavit of Nondisclosure for the project. Personal information (e.g., name, address) of participants will be collected for recruitment purposes, but on the datafile used for analyses, respondents will be identified by an ID number only. Within 48 hours of respondents’ participation in the usability testing session, the discussion notes will be edited, organized, and cleaned, and all direct identifiers will be stripped from the data set. All computer files used for this study work will be stored on secure, project-specific network drives and hard copies will be locked in secure locations (e.g., data will be in locked file cabinets within locked offices). Only staff working directly on the data analysis portion of the project will have access to the data files. Any presentations of data in reports will not include any portions of participants’ names, and once the final report is created, all personally identifiable information will be destroyed.

Estimate of Hour Burden

Recruitment of participants is estimated to take about 10 minutes per parent. It is expected that up to 35 parents will need to be contacted to form the desired group of 20 participants. It is possible that the programming of the parent survey may not be completed prior to the start of the usability tests, such that the full usability protocol cannot be implemented during the scheduled window; in this case, either a portion of the original parents will be rescheduled or, if needed, 10 additional parents will be recruited, to complete the usability testing later in 2019. Each usability test is expected to take 90 minutes. There is no cost to participants beyond the participation burden time. Table 1 provides the burden estimates for this study.

Table 1. Burden estimates for ECLS-K:2023 usability testing with parents

Respondent group	Number of respondents	Number of responses	Burden hours per respondent	Total burden hours
Parents – Recruitment	35	35	0.167	6
Parents ‒ Usability Testing	20*	20	1.5	30
Parents – Additional Sample Recruitment (if needed)**	10	10	0.167	2
Parents – Additional Usability Testing (if needed)**	5*	5	1.5	8
Total Burden	45	70		46

* Subset of initial contact group, not double counted in the total number of respondents.

** If the full usability testing protocol cannot be completed because the programming of the parent survey has not been finished in time to meet the initial schedule, parents who were initially recruited will be rescheduled to complete the testing at a later date. If needed, additional parents will also be recruited.

Project Schedule

The schedule of activities for the ECLS-K:2023 usability testing is provided in table 2.

Table 2. Schedule for the ECLS-K:2023 usability testing with parents

Activity	Tasks	Date ranges
Data collection	Recruit participants	September-October 2019
	Conduct in-person usability testing	October 2019
Analysis	Analyze data and produce report	November 2019
Report-out meeting	NCES and Westat meet to discuss findings and recommendations	by early December 2019

Cost to the Federal Government

The estimated cost to prepare for, administer, and report the results of the ECLS-K:2023 preschool parents usability testing is approximately $8,492. This cost includes salaried labor for Westat staff, participant incentives, and other direct costs associated with organization of the usability sessions.