Changes Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: July 23, 2019

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, NCES

FROM: Elise Christopher, NCES

SUBJECT: High School and Beyond 2020 (HS&B:20) Base-Year Full-Scale Study Recruitment and Field Test Update (OMB# 1850-0944 v.3).

The High School and Beyond 2020 study (HS&B:20) will be the sixth in a series of longitudinal studies at the high school level conducted by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education. HS&B:20 will follow a nationally representative sample of ninth grade students from the start of high school in the fall of 2020 to the spring of 2024 when most will be in twelfth grade. A field test will be conducted one year prior to the full-scale study. The study sample will be freshened in 2024 to create a nationally representative sample of twelfth-grader students. A high school transcript collection and additional follow-up data collections beyond high school are also planned. The NCES secondary longitudinal studies examine issues such as students’ readiness for high school; the risk factors associated with dropping out of high school; high school completion; the transition into postsecondary education and access/choice of institution; the shift from school to work; and the pipeline into science, technology, engineering, and mathematics (STEM). They inform education policy by tracking long-term trends and elucidating relationships among student, family, and school characteristics and experiences. HS&B:20 will follow the Middle Grades Longitudinal Study of 2017/18 (MGLS:2017) which followed the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011), thereby allowing for the study of all transitions from elementary school through high school and into higher education and/or the workforce. HS&B:20 will include surveys of students, parents, students’ math teachers, counselors, and administrators, plus a student assessment in mathematics and reading and a brief hearing and vision test. The HS&B:20 Base-Year Full-Scale study (BYFS) will begin in the fall of 2020. The request to conduct the HS&B:20 Base-Year Field Test (BYFT) and the BYFS sampling and state, school district, school, and parent recruitment activities, both scheduled to begin in the fall of 2019, was approved in June 2019 (OMB# 1850-0944 v.1-2). These activities include collecting student rosters and selecting the BYFS sample.

This request is to: (1) add instruction to the BYFT student roster template; (2) add response options to BYFT school administrator questionnaire; (3) revise existing and add new BYFT recruitment materials; and (4) finalize the BYFS student session length. Approval for the base-year full scale study data collection will be requested in a separate submission in early 2020.

The requested here changes do not affect the approved overall estimated respondent burden nor the total cost to the federal government for conducting this study.

(1) Add instruction to BYFT student roster template. During BYFT recruitment, we learned that some students at schools on a block schedule may not be taking a math course in the fall semester of their ninth-grade year. For that reason, we have added an instruction for schools to indicate “NA” if the student does not take a math course. This would help to distinguish between students who are not enrolled in a math course during the fall semester and students for whom the school neglected to provide teacher information.

(2) Add response options to BYFT school administrator questionnaire. Response options have been added to three questions on the school administrator instrument—HA0GRADESRV, HA0SPECVOTECH, HA0GRADSPCT:

1. After careful consideration, we added a response option of “ungraded” to the item asking what grades are served by the school (HA0GRADESRV). This response, which was included in OMB# 1850-0944 v.1 but then removed for OMB# 1850-0944 v.2, may or may not be required since eligibility criteria for HS&B:20 require schools to have a 9^th grade. However, we have since determined that although a school is in fact ineligible for participating in HS&B:20 if they do not have a 9^th grade, schools with 9^th grades may also have ungraded “grades” or ungraded students receiving instructions from that school; and while those grades and students will not be included in the sample, having ungraded “grades” and ungraded students is still a pertinent characteristic of schools in the sample. As a result, that option has been added back to the questionnaire.

2. In response to the question asking the percentage of ninth-grade students enrolled in a vocational, technical, or business program (HA0SPECVOTEH), we added a response option HA0SPECVTECHNA to allow respondents to indicate if the school does not have vocational, technical, or business programs.

3. To be comprehensive in response options to the question asking what students do after grade 12 (HA0GRADSPCT), we added the response option “Repeated 12th grade” (HA0GRADSREP12_LBL & HA0GRADSREP12). We also added a “Don’t know” (HA0GRADSDK_LBL, HA0GRADSDK) response option in case respondents had information available for some of the previous year’s 12^th-grade class but not all students in the class. In this way, the percentages reported should sum to 100 percent.

(3) Revise existing and add new BYFT recruitment materials. Five new materials have been developed for the HS&B:20 field test:

1. A school coordinator cover letter (Appendix A4b1) will be sent to the school coordinator in the package containing the previously approved coordinator checklist and parent permission materials.

2. In response to a request from at least one participating school requiring written assent from students for in-school data collection, a student assent form (Appendix A5g) was developed and added. This form will only be used in schools that require written student assent and will be collected and returned to RTI along with any written parental permission and refusal forms.

3. All students participating in HS&B:20 in-school will hear a student script (Appendix A5h) which will be read by the Session Facilitator to the students prior to the start of the session to provide information and instruction about the session and using the equipment. This script was inadvertently left out of the prior clearance request.

4. In addition to the new materials, language in the staff communication letters (Appendix A12a – A12c3) was revised to reflect a change in how we reference the certificate provided to school staff for their participation. While the certificate will be provided as originally planned, we will refer to the certificate as a professional development certificate from the U.S. Department of Education instead of a certificate for continuing education credits. This change in language is necessary to ensure that we are not overpromising what the certificate can be used for because not all school districts will accept the participation certificate for continuing education credits. A copy of the certificate is provided in Appendix A13a.

5. An incentive receipt form for the out of school data collection was also inadvertently omitted from the prior package and is included in Appendix A13b.

(4) Finalize the BYFS student session length. The approved HS&B:20 BYFS Recruitment and BYFT request (OMB# 1850-0944 v.1-2) included an experiment to be conducted during the field test recruitment that began in January 2019. The experiment was to offer two different student session lengths to determine whether the session length has an impact on school district and school decision-making regarding participation. Approximately half of the schools were asked to allow a 90-minute student session and half were offered a 45-minute student session. The approved experiment plan indicated that by early August 2019 we would submit an update indicating which session length will be offered to schools during the HS&B:20 BYFS recruitment.

Recruitment for the field test data collection is ongoing to achieve 75 participating schools. As of July 23, 2019, 66 schools have agreed to participate. For detailed results to date, please see the revised text shown under Part B section B.4 in this document. Given the results of the field test experiment to-date, during BYFS recruitment we plan to offer a 90-minute session to all schools and to offer a 45-minute session only as a refusal conversion strategy when needed.

The edits made to the last approved HS&B:20 clearance documents to reflect changes requested in this submission are listed below. Text marked in red font below reflects the additions and deletions that were made.

Part A

The following text was revised in section A.1.a (p.2):

In preparation for the HS&B:20 Base-Year Full-Scale study (BYFS), scheduled to take place in the fall of 2020, this in June 2019, the Office of Management and Budget (OMB) approved (OMB# 1850-0944 v.1-2) a request is to conduct the HS&B:20 Base-Year Field Test (BYFT) and the BYFS sampling and state, school district, school, and parent recruitment activities, both scheduled to begin in the fall of 2019. These activities include collecting student rosters and selecting the BYFS sample. Approval for the BYFS study data collection will be requested in a separate submission in early 2020. Approval for the base-year field test recruitment activities was received in December 2018 (OMB# 1850-0944 v.1).

This request is to: (1) add instruction to the BYFT student roster template; (2) add response options to BYFT school administrator questionnaire; (3) revise existing and add new BYFT recruitment materials; and (4) finalize the BYFS student session length.

The following text was deleted from section A.9 (Schools; pp.9-10):

With regards to when a 90- vs. 45-minute student session will be requested of schools during BYFS recruitment, based on the field test recruitment experience between January and August 2019, in August 2019 we will submit a change request describing the final plan for the BYFS student session length.

Part B

The following text was revised in section B.3 (School Recruitment, Determining the “right” amount of time to facilitate participation; p.12):

For BYFS, the 90-minute session will be offered to schools during recruitment and the 45-minute session will be offered if it results in significantly higher school participation than the 90-minute session in the field test. If the shorter session does not result in higher participation that is statistically significant, a 90-minute session will be offered with a 45-minute session used as only as part of a non-response and/or refusal conversion strategy. Based on the field test recruitment experience between January and August 2019, in August 2019 we will submit a change request describing the final plan for the BYFS student session length.

The following text was revised in section B.3 (School Recruitment, Public and Catholic School Recruitment; p.14):

The final plan for BYFS will depend on field test results and will be provided in August 2019 as a change request For BYFS, the 90-minute session will be offered to schools during recruitment and the 45-minute session will be offered only as part of a non-response and/or refusal conversion strategy.

The following text was revised in section B.4 (Field Test Experiments, School/Students; pp.16-17):

School/Students. Student data collection largely relies on the schools’ willingness to allow in-school data collection. A student survey that averages 90 minutes may require two class periods to administer. To examine the impact of reducing participation burden on the schools, an alternative design will fit the data collection within one class period (approximately 45 minutes), potentially gaining participation from a larger proportion of sample schools.

To better understand the effect of the length of the student session on school recruitment, the 309 schools in the field test sample will be were randomly assigned (after controlling for the same treatment within school district) to a 90-minute or a 45-minute in-school student administration request. For each condition, there will be was a sequence of nonresponse follow-up procedures that ultimately allows for out-of-school administration:

• Full student session: The initial request for schools will be was for a 90-minute in-school student administration. Schools and districts that refused, will be were asked to participate with a 45-minute administration. For those still refusing, a 90-minute out-of-school student administration will be was offered.

• Reduced student session: In this condition, the initial request to schools and school districts will be was for the reduced 45-minute student session administration, which will consists of demographic questions, the hearing and vision assessment, and the mathematics assessment. Students from schools that take took the 45-minute session in school will received a postcard asking them to complete the remainder of the session outside of school, specifically the student survey and reading assessment plus any math items they did not finish at school (see appendix A8c). Those schools refusing to participate in the shorter session will be were offered a 90-minute out-of-school student administration.

From the planned experiments, this one has the lowest statistical power as schools are the unit of analysis, i.e., we will should be able to detect only relatively large differences in participation. At the .05 level of significance and power of .80, we will should be able to detect an approximately 13 percentage point difference in school participation (e.g., 24 percent vs. 37 percent) with a one-tailed test. However, despite our school sample size limitations, we believe that such a difference from substantially reducing the burden on schools is possible and warrants a test.

Should If the field test recruitment effort yielded significantly higher school response rates with the 45-minute student session, we will would have implemented it in BYFS. If the differences are were not statistically significant, we can would plan to release the BYFS sample in two sample replicates, repeating the experiment in the first replicate in order to select the optimal student survey length for the second sample replicate. An alternative to this approach is was to use the 45-minute instrument as a nonresponse follow-up strategy for school-level (and school district-level) nonresponse. If the field test recruitment effort does did not yield higher school response rates with the 45-minute student session, the 90-minute session will would be offered to schools during BYFS recruitment, and the 45-minute session will would be used as a refusal conversion strategy.

As of July 23, 2019, 66 of the desired 75 schools have agreed to participate in the HS&B:20 BYFT. In preparation to begin school district and school recruitment in September 2019 for the Fall 2020 BYFS data collection, we reviewed the results of the experiment to date to determine whether there is a difference in participation among those schools offered a 90-minute session and those offered a 45-minute session. The 66 participating schools include five virtual schools whose students would necessarily participate outside of school. Thus, we looked to the remaining 61 schools to determine the outcome of the experiment, of which 33 were assigned the 90-minute session and 28 were assigned the 45-minute session. Similarly, we looked at the number of school districts that have declined to participate. Of the 17 districts that declined, eight were assigned to the 90-minute session (representing 28 schools) and nine were assigned to the 45-minute session (representing 21 schools). Though a small number of schools have expressly commented favorably on the one-class period session time, session time did not have a large impact on a school’s decision to participate in the study. We will therefore proceed with the option to offer the 90-minute session to schools during BYFS recruitment, and to use the 45-minute session only as a refusal conversion strategy. As in the field test, students participating in the 45-minute session in school will be asked to complete the remainder of the session outside of school (they will be asked to complete the BYFS student survey and reading assessment, but will not be asked to finish any math items they did not complete at school). Based on the field test recruitment experience between January and August 2019, in August 2019 we will submit a change request describing the final plan for the BYFS student session length.

Part C – No changes

Appendix A

The following new materials were added in Appendix A:

1. Appendix A4b1. School Coordinator Letter Accompanying Permission Materials (new) – page 10

2. Appendix A5g. Student Assent Form (new) – page 30

3. Appendix A5h. Student Script (new) – page 31

4. Appendix A13a. School Staff Participation Certificate (new) – page 92

5. Appendix A13b. Parent/Student Incentive Receipt Form (new) – page 93

The following sentence was revised in Appendices A12a through A12c.3 (pages 80-91) to clarify information about the certificate that school staff will receive for participating:

You may will receive continuing education credits a professional development certificate for your time from the U. S. Department of Education.

Appendix B

The following text was added to the “Student Math Teacher” column heading in Appendix B.1 (page 1) to provide instruction for documenting students who do not have a math course in the fall of 2019:

Student's Math Teacher **(if no math class taken by student during fall semester, please indicate by NA (not applicable) in Math Course Name Column. PLEASE DO NOT LEAVE BLANK

The following six response options (affecting three items—HA0GRADESRV pp.250-251, HA0SPECVOTECH pp.264-265, HA0GRADSPCT pp.266-267) were added to the administrator questionnaire (appendix B.5):

HA0GRADESRV (pp.250-251)

Question Wording:

[Our records indicate that this school serves the following grade(s). Please correct any grade(s) that this school does not serve. /Please tell us which grade(s) this school serves.]

(Check all that apply.)

Item Name: HA0GRADEPK
Item Wording: Pre-kindergarten

Item Name: HA0GRADEK
Item Wording: Kindergarten

Item Name: HA0GRADE1
Item Wording: 1st grade

Item Name: HA0GRADE2
Item Wording: 2nd grade

Item Name: HA0GRADE3
Item Wording: 3rd grade

Item Name: HA0GRADE4
Item Wording: 4th grade

Item Name: HA0GRADE5
Item Wording: 5th grade

Item Name: HA0GRADE6
Item Wording: 6th grade

Item Name: HA0GRADE7
Item Wording: 7th grade

Item Name: HA0GRADE8
Item Wording: 8th grade

Item Name: HA0GRADE9
Item Wording: 9th grade

Item Name: HA0GRADE10
Item Wording: 10th grade

Item Name: HA0GRADE11
Item Wording: 11th grade

Item Name: HA0GRADE12
Item Wording: 12th grade

Item Name: HA0GRADPST12
Item Wording: Any grade level higher than 12

Item Name: HA0GRADEUNGRD
Item Wording: Ungraded

HA0SPECVOTECH (pp.264-265)

Question Wording:
What percentage of the current 9th-grade class at this school is in a vocational, technical, or business program?

(Please round to the nearest whole number. Enter '0' if none.)

Item Name: HA0SPECVOTECH
Item Wording: ____%

Item Name: HA0SPECVTECHNA
Item Wording: Does not apply; this program is not offered at this school

HA0GRADSPCT (pp.266-267)

Question Wording:
What percentage of last year's (2018-2019) 12th-grade class at this school...
(Please round your responses to the nearest whole number. Enter '0' if none. Your responses should sum to 100.)

Item Name: HA0GRADS4YRPS_LBL
Item Wording: Went on to 4-year, bachelor's degree-granting colleges?

Item Name: HA0GRADS4YRPS
Item Wording:   ____%

Item Name: HA0GRADS2YRPS_LBL
Item Wording: Went on to 2-year, associate's degree-granting colleges or technical institutes?

Item Name: HA0GRADS2YRPS
Item Wording:   ____%

Item Name: HA0GRADSJOB_LBL
Item Wording: Entered the labor market?

Item Name: HA0GRADSJOB
Item Wording:   ____%

Item Name: HA0GRADSMIL_LBL
Item Wording: Joined the military without entering postsecondary education?

Item Name: HA0GRADSMIL
Item Wording:   ____%

Item Name: HA0GRADSREP12_LBL
Item Wording: Repeated 12th grade?

Item Name: HA0GRADSREP12
Item Wording:   ____%

Item Name: HA0GRADSOTHER_LBL
Item Wording: Did something else?

Item Name: HA0GRADSOTHER
Item Wording:   ____%

Item Name: HA0GRADSDK_LBL
Item Wording: Don't know

Item Name: HA0GRADSDK
Item Wording:   ____%

3