Volume 1 HS&B 2020 Cog Labs Round 2

National Center for Education Statistics (NCES)

Volume I

Supporting Statement

High School and Beyond 2020 (HS&B:20) Base-Year Cognitive and Usability Testing Round 2

OMB# 1850-0803 v.252

August 2019

Attachments:

Attachment I – Recruitment Procedures and Contacting Materials

Attachment II – Eligibility Screening Questions

Attachment III – Consent to Participate in Research

Attachment IV – Interview Protocol

Attachment V – Cognitive and Usability Interview Protocol: 9th-Grade Student Survey

Attachment VI – Cognitive and Usability Interview Protocol: Parent Survey

Attachment VII – Cognitive and Usability Interview Protocol: Teacher Survey

Attachment VIII – Cognitive and Usability Interview Protocol: School Administrator Survey - Principal

Attachment IX – Cognitive and Usability Interview Protocol: School Administrator Survey - Designee

Attachment X – Cognitive and Usability Interview Protocol: School Counselor Survey

Attachment XI – Cognitive and Usability Interview Protocol: English Language Screener

Attachment XII – Cognitive and Usability Interview Protocol: Math Assessment

Attachment XIII – Cognitive and Usability Interview Protocol: Spanish Speakers

Attachment XIV – Eligibility Screening Questions: Spanish Speakers

Attachment XV – Consent to Participate in Research: Spanish Speakers

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews, to improve methodologies, survey and test questions, and/or delivery methods.

The High School and Beyond 2020 study (HS&B:20) will be the sixth in a series of longitudinal studies at the high school level conducted by NCES. HS&B:20 will follow a nationally-representative sample of 9th-grade students from the start of high school in the fall of 2020 to the spring of 2024 when most will be in 12th grade. The study sample will be freshened in 2024 to create a nationally representative sample of 12th-graders. A high school transcript collection and additional follow-up data collections beyond high school are also planned. The NCES secondary longitudinal studies examine issues such as students’ readiness for high school; the risk factors associated with dropping out of high school; high school completion; the transition into postsecondary education and access/choice of institution; the shift from school to work; and the pipeline into science, technology, engineering, and mathematics (STEM). They inform education policy by tracking long-term trends and elucidating relationships among student, family, and school characteristics and experiences. HS&B:20 will follow the Middle Grades Longitudinal Study of 2017-2018 (MGLS:2017) which followed the Early Childhood Longitudinal Study, Kindergarten Cohort of 2011 (ECLS-K:2011), thereby allowing for the study of all transitions from elementary school through high school and into higher education and/or the workforce. HS&B:20 will include surveys of students, parents, students’ math teachers, counselors, and administrators. Students will also receive assessments in mathematics and reading and will be given a 10-minute hearing and vision test. Students designated as English Language Learners (ELLs), or who often speak a language other than English at home, will complete an English Language (EL) screener to confirm their ability to complete the student session in English. Approval for the HS&B:20 base-year full-scale recruitment activities and base-year field test, all scheduled to begin in the fall of 2019, was most recently received in August 2019 (OMB# 1850-0944 v.1-4).

The overarching purpose of HS&B:20 is to provide a rich variety of data to answer questions about how students’ backgrounds and school experiences affect education, employment, and other life outcomes. HS&B:20 represents a new and compelling evolution of NCES’s secondary longitudinal studies program. Many survey and test items planned for the upcoming HS&B:20 base-year field test have been previously tested or were included in prior NCES secondary longitudinal and other studies. While maintaining the hallmarks of prior study designs—longitudinal data collection; cognitive assessments measuring student achievement growth; contextual information from parents, teachers, and administrators; and administrative records collection—HS&B:20 also incorporates exciting innovations and further improvements.

This request is to conduct cognitive and usability testing starting in September 2019 in preparation for the HS&B:20 base-year full-scale study, allowing NCES to evaluate survey and assessment items that are either new or have been revised from items used in prior NCES longitudinal studies. Items to be evaluated include questions selected for each of the five HS&B:20 surveys: student, parent, administrator, teacher, and counselor. In addition, cognitive and usability testing will solicit feedback on a set of math assessment (test) items from 9th-graders and 12th-grade aged teenagers, as well as feedback on English Language screener items from non-native English-speaking high school students. Examples of items to be tested include a section asking students and parents about the parent’s employment; select items asking administrators about teacher demographics and certification; and a section asking counselors for information on dual enrollment programs. The full array of survey items to be tested is presented in Attachments V through XIII of this submission. Due to the proprietary nature of assessment items, Attachments XI through XIII contain the probes but not the full items to be used in the English Language screener and math assessment protocols. HS&B:20 will be administered in multiple modes to include web, Computer-Assisted Telephone Interviewing (CATI), and Computer-Assisted Personal-Interviewing (CAPI). The current cognitive and usability testing will use the programmed instruments. The results will guide potential survey and assessment modifications, which will be used to refine the base-year field test items in preparation for the full-scale data collection. These refined items will be submitted to OMB for review in early 2020 as part of the HS&B:20 base-year full-scale data collection request. RTI International has been contracted by NCES to collect HS&B:20 data; and EurekaFacts, LLC (hereafter referred to as EurekaFacts) serves as RTI’s subcontractor for aspects of HS&B:20 cognitive and usability testing.

Background

The cognitive and usability testing for which this clearance is requested will be used to refine the survey and assessment questions, maximize the quality of data collected, and provide information on issues with important implications for the survey and assessment design, such as the following:

• Identify whether participants can provide accurate data;

• Evaluate the extent to which terms in questions are comprehended;

• Update and add terminology when necessary;

• Examine the thought processes used to arrive at answers to survey and assessment questions;

• Determine appropriate response categories to questions;

• Identify sources of burden and participant stress;

• Evaluate the appropriate group (e.g., students vs. parents) to ask about topics;

• Assess the ease of navigating the surveys and assessments;

• Examine participants’ comfort with selecting/entering and changing responses;

• Scrutinize the helpfulness of direction screens and available tools/aids;

• Evaluate usability aspects of the landing page screen where principals will begin the school administrator survey; and

• Evaluate clarity and understanding of a portion of the school administrator instrument that principals can assign to a designee to complete on their behalf.

Design and Context

The purpose of this cognitive and usability testing is to evaluate the wording, language, difficulty-level, content, and usability of select survey and assessment items. The results will be used to revise the survey and assessment instruments so they are clear and can be easily understood by participants in the HS&B:20 base-year full-scale data collection. Testing will be conducted using a subset of items from the base-year field test instruments, which will be administered to students/teenagers, their parents, administrators, teachers, and counselors. More specifically, the testing sample will include 9th-grade students, 12th-grade students, 12th-grade aged teenagers no longer enrolled in high school (including those who have graduated early and those who have elected not to continue their formal education), and high school students who do not speak English as their first language. In addition, the sample will include the parents of the 9th-grade students participating in the cognitive and usability testing, 9th-grade math teachers, school administrators and their designees who work with high school students, and school counselors who work with high school students.

EurekaFacts staff have extensive experience in cognitive and usability testing methodologies and will be responsible for recruiting participants, conducting the interviews, compiling interview audio/video recordings, and summarizing the findings. Interviewer probing during testing will encourage participants to elaborate about their understanding of the HS&B:20 survey and assessment questions and how they formulate their answers. Participant responses to probes will be used to evaluate and revise question wording as needed. Interviewers will use a combination of generic probes and item-specific probes to elicit information on the mental steps participants took to arrive at an answer. Generic probes will be particularly useful when the participant exhibits signs of confusion or expresses difficulty with the question. Specific probes will be used to evaluate potential issues that are pre-determined (see Attachments V through XIII for the generic and specific probes). Probing also ensures that participants, especially teenagers, remain engaged in the process as they work on the items.

All students/teenagers and parents will complete their session at the EurekaFacts office. School staff participants (i.e., teachers, administrators, designees, counselors) will have the option to complete their session in-person, remotely via telephone, or virtually through an on-line conferencing service, such as Skype for Business. This approach provides the needed flexibility for accommodating this population and meeting geographic diversity requirements. Remote participation will require that participants have access to view the survey on a computer or tablet and have a strong and reliable phone service connection to speak with the interviewer. The EurekaFacts office can accommodate testing observers. For remote interviews, testing observers will be able to observe the interview through a Skype meeting invitation and to communicate with each other and the interviewer via a chat room. All interviews will be audio-recorded and may be video-recorded, and the recordings will be available to NCES and RTI’s HS&B:20 staff for review. EurekaFacts will compile and summarize their early observations once half of the interviews have been conducted and produce formal reports at the end of the testing.

The sample for testing will contain a diverse pool of respondents within respondent type. Specifically, this sample will include:

• Students/teenagers

  • 9th-grade students,

  • 12th-grade students,

  • 17- and 18-year-olds who are not enrolled in school,

  • High school students whose first language is other than English,

• Parents of the 9th-grade students,

• Math teachers currently teaching 9th-grade students,

• Principals and their school staff designees who work with high school students, and

• School counselors who work with high school students.

Recruitment will continue until a total of 155 participants have participated in testing (see Table 1). The student sample will include a mix of demographic characteristics, including race and ethnicity, gender, type of school attended, socio-economic status, and urbanicity. Student/teenager participants will be identified using EurekaFacts’ database of potential research participants in the Greater Washington, DC - Baltimore, MD Metropolitan area as well as referrals, advertisements in student newspapers and online forums, social media postings, flyers, and through outreach to community-based organizations and after-school programs serving students and their parents.

The school staff sample will include 9th-grade math teachers, school administrators and their designees, and school counselors. Participants will be selected from different types of schools (e.g., public, charter, and private) and from various geographic regions across the U.S. School staff will also be selected to represent a range of years working in the field of education. Designees will be recruited in pairs with the school administrators. School staff participants will be identified using EurekaFacts’ database of potential research participants across the United States, as well as referrals, advertisements in educational publications and online forums, social media postings, flyers, and through outreach to educational and professional organizations and unions serving educational professionals. Screening of all potential participants will be conducted using an online survey or in a telephone interview comprised of eligibility questions specific to the needs of this study to ensure that potential participants qualify (see Attachment II & XIV).

Table 1: Number of screening and testing participants by participant type

Participant type	Expected to be screened	Testing participants
9th-grade student/parent pairs (Including 9th-grade survey and math assessment)	333	50 (25 student-parent pairs)
12th-grade students/17- and 18-year-olds not in school	375	25
High school age teenagers who are non-native English speakers	375	25
Principals who work with high school students	225	15
Principals’ designees who work with high school students	150	10
School counselors who work with high school students	225	15
9th-grade math teachers	225	15
Study Total	1,908	155

Attachments I through IV and XIV present EurekaFacts’ interview protocol and materials, including recruitment materials, eligibility screening questions, consent forms, and scripts. Attachments V through X provide all survey cognitive interview items with generic probes and specific probes planned for each. Attachments XI through XII provide the probes to be used to assess the English Language screener and math assessment, and Attachment XIII provides the probes to be used to assess the Spanish speakers. To maintain the session time for interviews (outlined in Table 3), fewer survey/assessment questions may be asked of each participant than those presented in the attachments.

Estimated Participant Burden

To yield 155 completed interviews across the eight respondent types (outlined in Table 1), we anticipate screening up to 1,908 individuals to ensure that we are achieving the desired distribution of participant types. The screening process, on average, is estimated to take about 4 minutes per person (see Attachment II & XIV). Testing sessions will last between 30 and 90 minutes. Table 2 summarizes the anticipated burden, by length of interview.

Table 2: Estimated participant burden

Activity	Number of participants	Number of responses	Minutes per participant	Maximum total burden hours1
Screening	1,908	1,908	4	127
Cognitive interview
9th-grade students	25*	25	90	38
Parents of 9th-grade students	25*	25	60	25
12th-grade students/17- and 18-year-olds not in school	25*	25	30	13
High school age teenagers who are non-native English speakers	25*	25	30	13
9th-grade math teachers	15*	15	60	15
Principals who work with high school students	15*	15	30	8
Principals’ designees who work with high school students	10*	10	60	10
School counselors who work with high school students	15*	15	60	15
Study Total	1,908	2,063	N/A	264

* Subset of the screened group.

¹ Rounded to whole hours.

Estimate of Costs for Recruiting and Paying Participants

In order to be able to recruit a representative range of participants, and to thank them for their time and participation, we will offer prospective participants incentives for completing the cognitive interview. As in Round 1 of this testing (OMB# 1850-0803 v.240), the incentive amount and interview duration vary based on participant type, with incentives ranging from $25 to $50. Table 3 details the different incentive amounts and time commitments. Note that students/teenagers participating in 30-minute interviews will receive a $25 incentive, whereas 9th-grade students, who will be completing 90-minute interviews, will receive a $35 incentive to encourage them to participate in the longer session length.

Table 3: Incentive details by participant type

Participant Type	Number of participants	Length of time (minutes)	Incentive Dollar Value	Total Incentive Cost
9th-grade students	25	90	$35	$875
Parents of 9th-grade students (for interview)	25	60	$50	$1,250
12th-grade students/17- and 18-year-olds not in school	25	30	$25	$625
Parents of 12th-grade students/17- and 18-year-olds not in school (for transportation)	25	N/A	$25	$625
High school age teenagers who are non-native English speakers	25	30	$25	$625
Parents of high school age teenagers who are non-native English speakers (for transportation)	25	N/A	$25	$625
9th-grade math teachers	15	60	$50	$750
Principals who work with high school students	15	30	$50	$750
Principals’ designees who work with high school students	10	60	$50	$500
School counselors who work with high school students	15	60	$50	$750
Study Total	205	N/A	N/A	$7,375

Cost to Federal Government

The cost to the federal government for conducting these cognitive interviews will be $370,000 under the EurekaFacts subcontract to RTI. This cost includes recruitment, conducting interviews, analyses, report writing, and participant incentives.

Assurance of Confidentiality

Cognitive testing participants will be informed that their participation is voluntary and that:

EurekaFacts, LLC is carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

All participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials for each participant together. The participant ID will not be linked to the participant’s name. Participants will be sent a consent form (Attachment III & XV) via email, which they will need to sign, scan, and send back to EurekaFacts’ office in order to confirm their participation. The signed consent forms will be kept separately from the interview files for the duration of the study and will be destroyed three years after the final report is released.

Schedule for HS&B:20 Cognitive and Usability Testing

EurekaFacts will begin recruiting and screening potential participants upon receiving OMB clearance. Informed by the testing, a modified draft of the surveys, 9th-grade math assessment, and English Language screener will be used in the HS&B:20 base-year full-scale data collection which will begin in September 2020.

Activity	Dates
Recruit cognitive/usability test participants	September – December 2019
Conduct cognitive/usability testing of surveys	September – December 2019
Conduct cognitive/usability testing of math assessment and English Language screener	October 2019 – January 2020
Finalize revisions to item wording	January – February 2020

8