Part B HS&B 2020 BY Field Test Recruitment

Part B of this submission presents information on the collection of information employing statistical methods for the HS&B:20 base-year field test data collection.

B.1 Respondent Universe

The target populations for the HS&B:20 full-scale study consists of 9th grade students in public and private schools in the 50 United States and District of Columbia as of fall 2020¹. Excluded from the target universe will be special education schools, area vocational schools that do not enroll students directly, and schools associated with temporary housing such as correctional facilities and treatment centers.

The HS&B:20 field test will include 9th- and 12th-grade students and will be conducted during the fall of the 2019-20 school year in six geographic locations, metropolitan statistical areas (MSA), to ensure sampled schools come from each of four broad Census regions (Northeast, Midwest, South, and West). Recruitment is scheduled to begin in January 2019 and data collection in August 2019.

The field test will employ a two-stage sampling design, with schools selected in one stage, and then students selected within schools. Schools will be selected using simple random sampling within school sampling strata.

The primary sampling units (PSU) of schools will be selected from two databases of the U.S. Department of Education. The 2015-2016 Common Core of Data (CCD) will be used for selection of public schools while private schools will be selected from the 2015-2016 Private School Universe Survey (PSS). The secondary sampling units (SSU) of students will be selected from student rosters that will be secured from participating schools.

B.2 Procedures for the Collection of Information

HS&B:20 will collect data from high school students and their parents, math teachers, guidance counselors, and school administrators. Data will be collected from ninth graders in the fall of 2020 as they begin high school and again in the spring of 2024 when most students in the sample will be seniors at the end of their high school career. The field test data collections will be conducted one year prior to their full-scale counterparts. Collecting data at these timepoints from students, parents, teachers, counselors, and administrators, with high school transcripts collected after high school, will culminate in a rich data set that will provide educators, policymakers, and researchers with information about transitions, outcomes, and experiences in multiple contexts.

School Frames and Samples

RTI plans to use NCES’ latest Common Core of Data (CCD:2015–2016) as the public school sampling frame and Private School Survey (PSS:2015–2016) as the private school sampling frame. Given that these two sample sources provide comprehensive listings of schools, and that CCD and PSS data files have been used as school frames for a number of other school-based surveys, it is particularly advantageous to use these files in HS&B:20 for comparability and standardization across NCES surveys.

As mentioned earlier, the survey population for the full-scale study of HS&B:20 consists of all 9th graders in the 50 states and District of Columbia enrolled in:

regular public schools, including state department of education schools, that include 9th grade;
Bureau of Indian Education schools that include 9th grade; and
Catholic and other private schools that include 9th grade.

Excluded for this study will be the following:

schools with no 9th grade;
ungraded schools;
special education schools;
area vocational schools that do not enroll students directly;
non-domestic Department of Defense schools; and
closed public schools.

The base-year full-scale school sample will be selected using a stratified probability-proportionate-to-size (PPS) methodology for which a composite size measure methodology developed by RTI statisticians (Folsom, Potter, and Williams, 1987) will be used. This methodology will support the desired oversampling of students in key analytical domains (e.g., Asians and American Indian and Alaskan natives), maintain near equal sampling weights for students within each domain, and result in approximately equal total student sample sizes within sampled schools. Details of school sample selection for the field test are provided next.

Field Test School Sample

The base-year field test school frame will include 1,092² schools that report offering ninth and twelfth-grade instruction to at least 35 ninth- and 35 twelfth-grade students and are located within six MSAs. The school frame will also include 53 schools that report offering ninth-grade instruction to at least 35 ninth-grade students but do not offer instruction for twelfth grade.

Schools will be stratified by MSA and, within MSA, into two groups based on the presence or absence of twelfth-grade instruction: those schools offering ninth- and twelfth-grade instruction and those schools offering instruction in ninth but not twelfth grade. Schools offering both ninth- and twelfth-grade instruction will be further stratified into the following nine groups:

Catholic schools
Other private schools
Non-virtual magnet schools
Non-virtual/non-magnet charter schools
Virtual schools
Non-magnet/non-charter/non-virtual public schools in the Northeast
Non-magnet/non-charter/non-virtual public schools in the Midwest
Non-magnet/non-charter/non-virtual public schools in the South
Non-magnet/non-charter/non-virtual public schools in the West

Three hundred nine schools will be sampled and randomly assigned to receive an offer of a 90-minute student session or receive an offer for a single class period (e.g., 45-minute) student session.

The sample allocation was designed to produce 75 participating schools with approximately equal numbers of participating schools in each of six MSAs. The school strata and a sample allocation are shown in Table 1.

Table 1. HS&B:20 Base-Year Field Test School Sample Allocation

MSA	School Frame Count	Grade Level	School Type	Public School Type	School Sample Size	School Participation Goal
MSA	School Frame Count	Grade Level	School Type	Public School Type	School Sample Size	School Participation Goal
Total	1,145				309	75
A	13	9th but no 12th	Public	-	6	1
A	22	9th and 12th	Public	Charter not Virtual or Magnet	0	0
A	29	9th and 12th	Public	Magnet not Virtual	0	0
A	0	9th and 12th	Public	Virtual	0	0
A	40	9th and 12th	Public	non-magnet/non-charter/non-virtual – City	18	4
A	76	9th and 12th	Public	non-magnet/non-charter/non-virtual – Suburb	26	6
A	3	9th and 12th	Public	non-magnet/non-charter/non-virtual – Town	0	0
A	13	9th and 12th	Public	non-magnet/non-charter/non-virtual – Rural	10	2
A	15	9th and 12th	Catholic	-	0	0
A	7	9th and 12th	Other Private	-	0	0
B	1	9th but no 12th	Public	-	0	0
B	31	9th and 12th	Public	Charter not Virtual or Magnet	0	0
B	83	9th and 12th	Public	Magnet not Virtual	10	2
B	1	9th and 12th	Public	Virtual	0	0
B	2	9th and 12th	Public	non-magnet/non-charter/non-virtual – City	0	0
B	18	9th and 12th	Public	non-magnet/non-charter/non-virtual – Suburb	18	8
B	0	9th and 12th	Public	non-magnet/non-charter/non-virtual – Town	0	0
B	1	9th and 12th	Public	non-magnet/non-charter/non-virtual – Rural	1	1
B	15	9th and 12th	Catholic	-	0	0
B	29	9th and 12th	Other Private	-	10	2
C	0	9th but no 12th	Public	-	0	0
C	5	9th and 12th	Public	Charter not Virtual or Magnet	0	0
C	2	9th and 12th	Public	Magnet not Virtual	0	0
C	8	9th and 12th	Public	Virtual	6	1
C	10	9th and 12th	Public	non-magnet/non-charter/non-virtual – City	10	5
C	6	9th and 12th	Public	non-magnet/non-charter/non-virtual – Suburb	6	4
C	2	9th and 12th	Public	non-magnet/non-charter/non-virtual – Town	0	0
C	1	9th and 12th	Public	non-magnet/non-charter/non-virtual – Rural	0	0
C	3	9th and 12th	Catholic	-	0	0
C	2	9th and 12th	Other Private	-	0	0
D	23	9th but no 12th	Public	-	6	1
D	43	9th and 12th	Public	Charter not Virtual or Magnet	10	2
D	20	9th and 12th	Public	Magnet not Virtual	0	0
D	2	9th and 12th	Public	Virtual	0	0
D	26	9th and 12th	Public	non-magnet/non-charter/non-virtual – City	10	2
D	117	9th and 12th	Public	non-magnet/non-charter/non-virtual – Suburb	18	4
D	3	9th and 12th	Public	non-magnet/non-charter/non-virtual – Town	3	1
D	15	9th and 12th	Public	non-magnet/non-charter/non-virtual – Rural	10	2
D	30	9th and 12th	Catholic	-	10	2
D	27	9th and 12th	Other Private	-	0	0
E	6	9th but no 12th	Public	-	6	1
E	42	9th and 12th	Public	Charter not Virtual or Magnet	0	0
E	0	9th and 12th	Public	Magnet not Virtual	0	0
E	1	9th and 12th	Public	Virtual	0	0
E	45	9th and 12th	Public	non-magnet/non-charter/non-virtual – City	22	5
E	40	9th and 12th	Public	non-magnet/non-charter/non-virtual – Suburb	18	4
E	5	9th and 12th	Public	non-magnet/non-charter/non-virtual – Town	5	1
E	12	9th and 12th	Public	non-magnet/non-charter/non-virtual – Rural	10	2
E	6	9th and 12th	Catholic	-	0	0
E	8	9th and 12th	Other Private	-	0	0
F	10	9th but no 12th	Public	-	6	1
F	15	9th and 12th	Public	Charter not Virtual or Magnet	0	0
F	39	9th and 12th	Public	Magnet not Virtual	0	0
F	0	9th and 12th	Public	Virtual	0	0
F	24	9th and 12th	Public	non-magnet/non-charter/non-virtual – City	14	3
F	67	9th and 12th	Public	non-magnet/non-charter/non-virtual – Suburb	18	4
F	9	9th and 12th	Public	non-magnet/non-charter/non-virtual – Town	9	2
F	28	9th and 12th	Public	non-magnet/non-charter/non-virtual – Rural	13	2
F	16	9th and 12th	Catholic	-	0	0
F	28	9th and 12th	Other Private	-	0	0

The first step in the sampling process involves the selection of two non-magnet/non-charter/non-virtual schools with certainty to ensure that at least two schools are sampled from public-school districts that require a research application. After selection of these schools, the school sample size associated with the school sampling strata for those selected schools will be reduced by two. The second step in the sampling process involves the selection of the single Bureau of Indian Education (BIE) school with certainty. After selection of this BIE school, the school sample size associated with the school sampling strata for that selected school will be reduced by one. The third step in the sampling process involves the use of simple random sampling to select 306 more schools. Schools will be randomly assigned to be offered a single class period (e.g. 45-minute) or a 90-minute student session in such a fashion as to ensure that schools in the same district or diocese will be assigned the same offer. For recruitment efficiency and power associated with recruitment experiments, the full sample of 309 schools is planned to be released for recruitment all at once. The sample of 309 schools is expected to yield 75 or more participating schools. The recruitment team will continue to encourage participation for the purposes of the field test in-school session length experiment (described further in B.3) even after the 75-school yield target is reached. If considerably more than 75 schools agree to participate, a subset of these schools may be informed that they will not need to take part.

Field Test Student Sample

Within participating schools, students will be stratified into grades 9^th or 12^th, and a simple random sample of 35 ninth-grade students will be selected and, for those schools that offer twelfth-grade instruction, a simple random sample of 35 twelfth-grade students will be selected. The desired yield for the field test is 2,120 students enrolled in grade 9 and 2,007 students enrolled in grade 12. An estimated 5 percent of sampled students are assumed to be ineligible and 85 percent of eligible students are assumed to participate. Thus, to achieve a yield of 2,120 ninth-grade students, the parents of approximately 2,625 ninth-grade students will need to be contacted for consent (2,625*0.95*0.85=2,120). Similarly, the parents of approximately 2,485 twelfth-grade students will need to be contacted for consent (2,485*.95*.85=2,007).

School Recruitment Approach

Gaining cooperation from school districts and schools is paramount to the success of this voluntary study. However, recruitment efforts in similar studies have been meeting with increasing challenges that must be carefully mitigated to ensure adequate school participation. For example, in 1998–99 the Early Childhood Longitudinal Study had a weighted school-level response rate of 74 percent,³ whereas 12 years later, the successor ECLS-K:2011 cohort study had a weighted school-level response rate of 63 percent.⁴ Additionally, response rates tend to be lower for schools that serve older students (e.g., the High School Longitudinal Study of 2009 (HSLS:2009) had a weighted school-level response rate of 56 percent,⁵ and the Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) had an unweighted school-level response rate of 39 percent).

As described in section B.4 of this document, to better understand the effect of the length of the student session in terms of school recruitment, the HS&B:20 will split the sample of schools being recruited and request for half to participate in a study with a 90-minute student session and for the other half in a study with a 45-minute student session. For more detail, see section B.4.

Endorsements. Support from leading education organizations can, at times, be influential to school districts’ and schools’ decision to participate. Prior to contacting sampled school districts or schools, we will request the endorsement and support from relevant organizations and key stakeholders in secondary education (see appendix A.1). Organizations will be able to provide a letter of support and/or electronic endorsement of the study. Logos from endorsing organizations will be included on the HS&B:20 study website.

State Endorsement. As part of our study outreach efforts, all states will be contacted to inform them that the study will be taking place. States will be asked to provide a letter of endorsement to encourage school districts’ and schools’ participation in the study should schools in their state be selected. Letters will be sent to the state superintendent with copies to the state-level director of research and/or director of secondary education, as applicable (see appendix A.2). Once the sample schools are selected, senior HS&B:20 recruitment staff will contact state staff with schools in the sample to discuss the study and secure support. Endorsement letters received by the state are included in all mailings to districts and schools within the state.

School District and Diocesan Notification and Recruitment. Once states have been contacted, whether an endorsement letter was received or not, school districts and dioceses that do not require a research application will be notified that schools in their district have been selected for the study. The letter to school districts will state that NCES’s contractor, RTI International, will contact the school within two weeks and that they may contact RTI with questions (see appendix A.3).

Research applications will be prepared for any school districts that require the approval of applications in order to conduct research in schools in their jurisdiction. If a school district notifies us that an application must be submitted, or some other requirement must be fulfilled, study staff will be prepared to respond to such requirements. If a district chooses not to participate, all reasons will be documented to help formulate a strategy for refusal conversion attempts. Participating districts may be asked to provide student roster information on the school’s behalf to reduce the burden on the school.

Public and Catholic School Recruitment. Two weeks after the district was notified, or after the district provides approval when required, recruitment will commence at the school-level. Schools will receive a colored folder that contains a letter (appendix A.4a), study information sheet (appendix A.6a), frequently asked questions (FAQs) about the study (appendix A.6b), and a study brochure (appendix A.6c). Materials will be sent via overnight delivery and follow-up will occur within three business days. The first contact will be intentionally assigned based on prior history of working with the schools and school or district characteristics. First contacts may include modes such as a telephone call from recruitment staff, study management staff at RTI, or NCES staff; or an in-person visit to the school. Each of these modes, as well as email communication, may be used throughout the recruitment process as needed.

Once a school agrees to participate in HS&B:20, a recruiter will work with the school to name a member of the school’s staff to serve as the school coordinator for the study. The recruiter will work with the school coordinator to schedule study activities at the school, including gathering student rosters, distributing consent materials to parents of sample students, and arranging the session logistics. Roster instructions will be sent electronically in the fall of 2019 (see Appendix A10) and are also available on the website (see Appendix A7a). If a school is experiencing difficulty with preparing the roster, the district may be asked to provide the roster on the school’s behalf.

In early communications, the recruiter will also gather information about the school including: what type of parental consent procedures need to be followed at the school; hours of operation, including early dismissal days, school closures/vacations, and dates for standardized testing; and any other considerations that may impact the scheduling of student sessions (e.g., planned construction periods, school reconfiguration, or planned changes in leadership). The HS&B:20 study recruitment team will meet regularly to discuss recruitment issues and develop strategies for refusal conversion on a school-by-school basis.

As mentioned, half of the schools will be presented with a 90-minute student session. The other half will be presented with a 45-minute (one-class-period session) student session. As a refusal conversion effort, schools assigned to the 90-minute session will be asked to consider a 45-minute instead. If the school still declines, an out-of-school student session will be offered. Similarly, as a refusal conversion effort, if a school that is offered the 45-minute student session declines, that school will also be offered an out-of-school student session.

Private and Charter School Recruitment. If a private or charter school selected for the base-year field test operates under a higher-level governing body such as a diocese, a consortium of private schools, or a charter school district, we will use the district-level recruitment approach with the appropriate higher-level governing body. If a private or charter school selected for the field test does not have a higher-level governing body, the school recruitment approach outlined above will be used.

Roster Collection. Beginning in the fall of 2019, a roster of all ninth- and twelfth-grade students will be requested (appendix A.10). The rosters may be provided from the district or from the school, and it will be requested that the roster be provided once the enrollment for the school year has stabilized (which is often approximately 4 weeks into the school year) to increase accuracy. Key information needed for student sampling will be requested, such as: student name; school or district student ID number; date of birth; grade level; gender; race/ethnicity; and ELL status. Each of these characteristics is important for sampling purposes, but we will work with schools that are unable to provide all of the information to obtain the key information available. Based on this information, the student sample will be drawn. As part of the roster collection, the study will also request from the school coordinator or designated district personnel the following information for each student eligible for sampling: student’s parent and/or guardian contact information (e.g., mailing address; landline phone number; cell phone number; e-mail address) and student’s math teacher. Schools and districts often find it easier, and therefore more efficient, to supply all of the desired information one time for all of their students. However, should it be problematic for any school or district to provide the parent and teacher information on the complete roster, the recruitment team will gather that information as a second step for the sampled students only. If the school and/or district is unwilling to provide parent contact information for the sampled students, the team will work with the school and/or district to determine the best way to contact parents (e.g., the school coordinator or designated district personnel would facilitate contacting parents and/or would mail the required materials to parents using the contact information they have on file). Parent contact information is required to conduct the out-of-school student data collection.

The roster request will include a template and secure transfer options to deliver the rosters. The data quality of the student rosters will then be evaluated by:

reviewing and assessing the quality and robustness of student and parent information available at each school, including contact information for parents;
reviewing and assessing the quality of the data on student-teacher linkages;
addressing any incompleteness or irregularities in the roster file;
requesting additional information as needed from the school coordinator or designated district personnel; and
(re)verifying that the sampled students are currently in attendance in the school.

Parent Recruitment. For schools allowing in-school student sessions, schools will be given the option of one of three types of parental permission letters: notification (appendix A.5a), implicit permission (opt out) (appendix A.5b), or explicit permission (opt in) (appendix A.5c). Each type of consent requires that parents be notified that their children have been selected for the study. With a notification letter, no permission form is sent home since no action is required on the part of the parent. For implicit consent (opt out), the school does not require verbal or written consent for a student to participate in the study – parents are asked only to notify the appropriate person if they do not want their child to participate. With explicit consent (opt in), children may participate only if their parents provide written or oral consent for their children to do so. Proactive parent recruitment will be focused on maximizing the number of parents (1) returning signed explicit consent forms and (2) completing the parent survey. Because implicit consent does not require a verbal or written response from parents, these parents will not be contacted about consent forms. The letter accompanying the parent permission form will let parents know that the students will complete a survey, the math and reading questions, and a hearing and vision assessment. Parents will be told that they will receive results from the hearing and vision assessments.

The letters will be sent to the school for distribution to sampled students. Students in explicit permission schools will be offered a pizza party or equivalent for those who return the form by a designated date, regardless of whether permission is granted.

For schools that only permit out-of-school data collection, all contacts with the student will be conducted through the parent. The parent will receive a letter and an enclosed envelope with study information and study instructions. The parent will be asked to give the student the enclosed envelope with study information and study instructions. By giving the envelope to the student, it is implied that the parent consents to the child’s participation. The study information and instructions will include a student letter and the URL and login information for the student session. Students participating outside of school will not complete the hearing or vision assessment.

Parents will also receive an invitation to participate in the parent questionnaire (appendix A.11) at the start of data collection, with parent cases being added to the data collection process on a flow basis as parent contact information is provided by the school or parents provide such information on consent forms. Parent data collection will entail web-based self-administration with nonresponse follow-up by computer-assisted telephone interviewing (CATI). Letters inviting parents to participate in the study will contain a message on the envelope such as “Important, please open!”, “Your input is requested”, or “Help improve education! Open to find out how.”

Data Collection Approach

The HS&B:20 data collection will consist of a student session (survey, math and reading assessment, and hearing and vision assessment) as well as surveys for students’ parents, math teachers, guidance counselors, and school administrators. The field test data collection approach will be detailed in a subsequent clearance request submission but is summarized here as it will be described to schools during the recruitment process.

Once schools agree to participate, the designated school coordinator will be asked to provide a roster of all students in grades 9 and 12. The school coordinator will receive a template of the roster (see appendix B) with instructions to prepare and upload the roster electronically to the secure study website (see appendix A10). Upon receipt of the roster, RTI statisticians will randomly select about 35 students each from grades 9 and 12. About a month prior to the scheduled student session, a student tracking form listing the selected students will be sent to the school along with parent permission forms to distribute to the students.

Students will be asked to complete either a 90- or 45-minute session in a group administration in their school. The student surveys and direct assessments will take place in the school setting and be administered using Chromebooks (tablet-like computers with touchscreen capability and an attached keyboard) brought in to the school by HS&B:20 staff. HS&B:20 staff will also bring the necessary equipment for the hearing and vision assessments. This portion of data collection is referred to as the student session. To administer the survey and direct assessment in schools, study staff will work with schools to identify and utilize locations for administration that minimize distractions for the student and disruption to the school routine. Students will be prompted on screen to use the hearing and vision equipment that is at their desk to complete that portion of the session. This component may be turned off if schools decline to have the students’ vision and hearing tested.

Schools that decline to participate in the in-school session will be asked to allow students to participate outside of school. Students from these schools will be contacted through their parent using contact information provided by the school (appendix A.8a). The parent will be asked to give an envelope to the sampled student which contains a letter including login information for the student to complete the session online (appendix A.8b). Students who participate outside of school will not be asked to complete the hearing and vision assessments.

The parent (appendix A.11) survey will have an internet option and a telephone option, while the mathematics teacher (appendix A.12a), school counselor (appendix A.12b), and school administrator (appendix A.12c) surveys will be self-administered via the Web.

B.3 Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse

Recruitment

Methods to secure cooperation of school districts and schools, maximize response rates, and deal with nonresponse are described in this section.

Maximizing School Participation. The success of HS&B:20 hinges on securing the cooperation and maximizing response rates among school districts and schools, and then their students, parents, and staff. Participation among school districts and schools has been declining for voluntary school-based studies. Often, district and school personnel understand the value of the research but that does not offset their reasons for not participating. Reasons cited for not participating in voluntary school-based studies such as MGLS:2017 and HSLS:2009 are the high burden associated with participating, over-testing of students, loss of instructional time, lack of parent and teacher support, increased demands on school staff, and a moratorium on outside research. To mitigate these concerns, HS&B:20 has developed a recruitment plan to maximize school participation that is comprehensive and flexible in its approach and incentive structure to effectively secure cooperation for the study. Strategies recommended to maximize school participation include:

Outreach. Study and NCES name recognition add validity to the study when recruiting school districts and schools. Even prior to drawing the sample, outreach activities will be conducted to announce the upcoming study and begin to garner support from states, districts, schools, and stakeholders. Outreach activities will include:

Contacting school districts that are typically challenging to recruit to discuss their decision making process about participating in research studies and the benefits they would like to see from participating. This would be done generically as an exercise to help plan the recruitment effort prior to selecting the sample.
Distribution of a brief video to stakeholders, state, school districts, and schools to explain the importance of the study (Appendix A.6e).
A webinar explaining the importance of the study and the impact of the data collected. The webinar will be live, but a recording will be made available on the study website.
Attendance at conferences attended by stakeholders to introduce and promote the study.

Compelling recruitment materials. School districts and school staff are busy. Materials sent to these contacts must be informative, compelling, and brief. In addition, multiple types of materials (e.g., mailings, video, website) will be made available to ensure that decisionmakers have options to receive the message in the manner that works best for them. The materials will all be available on the study website for easy access. Reviewing these study materials should provide districts and school administrators with an understanding of the study’s value, the importance of HS&B:20, and the data collection activities required as part of the study. A full understanding of these factors will be important both to obtain cooperation and to ensure that schools and districts accept the data collection requests that follow.

Determining the “right” amount of time to facilitate participation. One of schools’ primary concerns about participating in outside research is the loss of instructional time. In an effort to balance the need to collect information with the desire to minimize the burden on schools and students, we are testing two student session lengths. Described in section B.4, half of the schools will be assigned at the outset to a 45-minute session and half to a 90-minute session. If a school in the 90-minutes group is unable to accommodate the full 90-minutes, it will be offered to partake in a one-class period or 45-minute session. Schools unable to provide a 45-minute session will be asked, as a refusal conversion option, to provide roster information so that students may be contacted to participate outside of school.

For teachers and parents, in the field test, we will also experiment with two survey lengths to determine which is most effective in securing participation. Teachers will be offered either the full survey or an abbreviated survey. The mathematics teacher survey will consist of three sections: the teacher-level portion of the survey will take either 15 or 8 minutes, the classroom portion will be a fixed 5 minutes, and the student-level report will be either 5 or 2 minutes per student. . Because of the small school sample size, no experimentation is being explored with guidance counselors or administrators beyond the student session length.

Varied communication modes. Prior to the start of recruitment, study team members will review the sample of school districts and schools to determine the appropriate mode of communication for each. Staff who may have connections to a particular area or district may be called upon to make the first contact. That staff person may remain the primary contact to the school district or school, or they may turn the school over to a recruiter for the collection of logistical information. Communication may be conducted via mail, email, phone, and in-person methods.

Leveraging participation in past studies. NCES has been conducting the longitudinal studies in schools since the early 1970s. HS&B:20 will leverage the participation information of school districts and schools from prior studies. Information gathered will include whether or not the school district or school participated, whether a research application was required, who in the district or school made the decision about participating, and the reasons for refusal, if applicable. This information will be used only to strategize who to contact and how to respond to previous concerns, if any, to encourage participation in HS&B:20.

Geospatial Modeling. The use of geospatial modeling helps overcome a serious challenge in base-year studies – a lack of information on sample members to use in predicting response. We will use geospatial modeling with data from open-access sources (e.g., NCES’s Education Demographic and Geographic Estimates, FBI’s Uniform Crime Reporting statistics) and RTI’s Enhanced Address-based Sampling Frame (Enhanced ABS Frame) to generate a superset of covariates that may help in estimating the likelihood of participation at all levels. From this covariate superset, we will identify a subset of substantive covariates that are significant predictors of response. Model coefficients from this geospatial model will then be used to predict, a priori, the likelihood of response for each unit in the HS&B:20 sample if the model provides sufficient ability to predict nonresponse. Interventions will be implemented based on the patterns of information found in the model. If we see that certain areas are underrepresented, we will focus our outreach efforts in those areas.

Volunteer hours for students. Many high schools require volunteer hours to be completed prior to graduation. As a token of appreciation for student participation, the U.S. Department of Education will provide a certificate to each participating student to acknowledge 2 hours of volunteer service for participation in the study. For confidentiality purposes, no study-specific information will be included on the certificate.

Flexible incentive package. As observed across NCES studies, schools and sample members vary in their motivation to participate in voluntary research, and what incentivizes some does not work for others. We are thus offering incentive choices to enable schools and sample members to determine what works best for them. The incentive structure is presented in section A.9 of the Supporting Statement Part A. To encourage schools to participate, we are also exploring offering school-related incentives to students and parents, such as entry to a school event or credit towards, or an item from, the school store. Such incentives further support the schools and can be seen as a benefit to both the school and students. Furthermore, increased flexibility in the items that are offered may encourage participation.

Continuing education or professional development credits for teachers, counselors, and administrators. In some states, districts, and schools, the school staff are required to complete continuing education credits or participate in documented professional development activities. We are exploring the requirements in each state to be able to offer continuing education (or professional development) credits to participating school staff.

Avoiding refusals. HS&B:20 recruiters will be trained to avoid direct refusals by focusing on strategies to solve problems or meet obstacles to participation faced by district or school administrators. They will endeavor to keep the door open while providing additional information and seeking other ways to persuade school districts and schools to participate.

B.4 Tests of Methods and Procedures

Full-scale HS&B:20 will collect base-year data from a sample of ninth-grade students in the fall of 2020, their teachers, their parents, their guidance counselors, and their school administrators. For the field test, we plan a limited set of experiments to refine the full-scale data collection procedures for the school/students, teachers, and parents. Experiments are not recommended for counselors or administrators due to the small school sample size. The experiments manipulate the amount of information requested and in what setting it is collected, as well as variations on how to incentivize participation.

School/Students. Student data collection largely relies on the schools’ willingness to allow in-school data collection. A student survey that averages 90 minutes may require two class periods to administer. To reduce the burden on the schools, an alternative design will fit the data collection within one class period (approximately 45 minutes), potentially gaining participation from a larger proportion of sample schools. The 309 schools in the field test sample will be randomly assigned (after controlling for the same treatment within school district) to a 90-minute or a 45-minute in-school student administration request. For each condition, there will be a sequence of nonresponse follow-up procedures that ultimately allows for out-of-school administration:

Full student session. The initial request for schools will be for a 90-minute in-school student administration. Schools and districts that refuse, will be asked to participate with a 45-minute administration. For those still refusing, a 90-minute out-of-school student administration will be offered.
Reduced student session. In this condition, the initial request to schools and school districts will be for the reduced 45-minute student session administration. Students from schools that take the 45-minute session in school will receive a postcard asking them to complete the remainder of the session outside of school (see appendix A8c). Those schools refusing to participate in the shorter session will be offered a 90-minute out-of-school student administration.

From the planned experiments, this one has the lowest statistical power as schools are the unit of analysis, i.e., we will be able to detect only relatively large differences in participation. At the .05 level of significance and power of .80, we will be able to detect an approximately 13 percentage point difference in school participation (e.g., 24 percent vs. 37 percent) with a one-tailed test. However, despite our school sample size limitations, we believe that such a difference from substantially reducing the burden on schools is possible and warrants a test.

Teachers. Teachers have multiple job responsibilities and demands on their time. Reducing the time demand to provide survey data has the potential to be a highly effective design feature to increase participation. Teachers within each participating school will be randomly assigned to provide the full teacher background information and the full teacher-student report, or to provide a reduced version of each set of data.

Assuming 75 participating schools, an average of 3.5 math teachers per school, and a teacher response rate of 75%, at the .05 level of significance and power of .80, we expect to detect a difference of 12 percentage points with a one-tailed test (the reduced burden is not expected to lead to a lower response rate).

Parents. Although the parent survey will not be administered in an institutional setting, the length of the survey may still be an important factor in gaining participation. In addition, there are multiple ways to incentivize parent participation. We plan to offer a total of $20 to each sample parent. Part of this $20 will be offering a $5 prepaid incentive. A prepaid incentive can be an effective way to encourage participation, with the remainder provided upon completion of the survey. However, we will examine the most effective way to provide the remaining $15 of the total $20 incentive, in monetary or non-monetary options.

The parents will be the largest sample, allowing for a test with more than two conditions. We plan on a 2x2 factorial design, with a treatment on survey length and a treatment on incentive format. Sample parents will be randomly assigned to a 30-minute survey or to a 15-minute survey. All parents will receive the $5 prepaid incentive. Then, the survey time treatment (30 minutes versus 15-minutes) will be crossed with assignment to be offered $15 in cash (or check) upon completion, or to choose a different incentive with a value of approximately $15, such as: school tickets to ball game, donation to the school for their participation, college preparation materials, movie tickets for family (2), a family board game, or donation to a charity.

The key interest is in the main effect of each treatment. Assuming 2,625 sample parents of 9^th grade students and an expected parent response rate of 50% (most conservative estimate), at the .05 level of significance and power of .80, we expect to detect a difference of 3.5 percentage points in the parent response rate.

B.5 Reviewing Statisticians and Individuals Responsible for Study Design and Conduct

The following individuals at the National Center for Education Statistics (NCES) are responsible for HS&B:20: Elise Christopher, Gail Mulligan, Chris Chapman, and Marilyn Seastrom. The following individuals at RTI are responsible for the study: Dan Pratt, Debbie Herget, Donna Jewell, David Wilson, and Laura Fritch, along with subcontractor staff Marc Berger and Rick Morgan (ETS).

References

Folsom, R.E., Potter, F.J., Williams, S.R. (1987) Notes on a Composite Size Measure for Self-Weighting Samples in Multiple Domains, Research Triangle Institute. http://ww2.amstat.org/sections/srms/Proceedings/papers/1987_141.pdf

1 While the HS&B:20 base-year full-scale sample will include only 9th grade students, the base-year field test sample will include both 9th and 12th grade students to prognosticate the progression that will be observed when reassessing the sample 9th grade students three years later, when most will be in 12th grade.

2 One of the 1,092 schools is a Bureau of Indian Education (BIE) school and, though no enrollment counts are provided by BIE schools to the CCD, BIE schools were included in the sampling frame as long as they reported offering instruction in grade 9 and grade 12.

3 Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G., Hagedorn, M.C., Daly, P., and Najarian, M. (2001). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), User’s Manual for the ECLS-K Base Year Public-Use Data Files and Electronic Codebook (NCES 2001-029). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

4 Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G., Hagedorn, M.C., Daly, P., and Najarian, M. (2012). Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011), User’s Manual for the ECLS-K:2011 Kindergarten Data File and Electronic Codebook (NCES 2013-061). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

5 Ingels, S.J., Pratt, D.J., Herget, D.R., Burns, L.J., Dever, J.A., Ottem, R., Rogers, J.E., Jin, Y., and Leinwand, S. (2011). High School Longitudinal Study of 2009 (HSLS:09). Base-Year Data File Documentation (NCES 2011-328). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-20

Part B HS&B 2020 BY Field Test Recruitment