SS Part B- 1830-NEW Roads to Success in North Dakota

This section describes the respondent universe for this study and the sampling and statistical methodologies proposed. This section also addresses suggested methods for maximizing response rates and for tests of procedures and methods, and introduces the statisticians and other technical staff responsible for design and administration of the study.

B.1 Respondent Universe and Sampling Methods

The target universe for the study includes all North Dakota public high schools and Bureau of Indian Education (BIE) schools with both an 11^th and 12^th grade. Schools with very small grade enrollments (fewer than 10) will be excluded from the universe and sampling frame due to the fact that many of these schools provide instruction in ungraded classes, and therefore differentiated 11^th- and 12^th-grade RTS instruction would not be possible.

According to the ND Educational Directory for 2013-2014, there are 167 public schools with a 12^th and 11^th grade in North Dakota (33 schools with just grade 9-12, and 134 with a combination of high school and lower grades). There are 7,348 12th-grade students and 7,414 11th-grade students. Assuming about 75 percent of schools have not implemented RTS in any grade (based on communication with ND officials), about 125 schools (containing about 5,511 12^th graders and 5,560 11^th graders) are potentially eligible for the study, with an average 12^th-grade and 11^th-grade class of 44 students (each grade) per school. With about 30 of these 125 schools excluded due to size, approximately 95 schools and 10,600 students will be eligible for the study.

The school sampling frame will come from the North Dakota Department of Public Instruction, and North Dakota CTE staff and regional CTE coordinators will assist in identifying, in advance, which schools are currently implementing RTS or have implemented RTS in recent years (so that, for example, schools who have provided some RTS programming to grade 10 or 11 students are not included in the study). To prepare for the contingency of encountering schools that have implemented RTS but are unknown to state staff and thus to the research team, our initial contact with schools will screen for prior RTS provision.

B.2 Procedures for Collecting Information

A randomized design will be employed in which target schools are assigned to the treatment or control conditions and all 11^th- and 12^th-grade students in a treatment school provided with RTS instruction. This design is necessitated by the relatively small size of schools in North Dakota. It also provides the advantage that cross-condition contamination within a school will not be possible.

To achieve adequate statistical power, the optimal number of schools for the study would be between 64 and 80 participating schools, depending on various assumptions. However, for maximum statistical power and to better account for the possibility of some schools refusing to participate or dropping out, all 95 estimated eligible schools (125 schools minus the 30 schools with very small 11^th- and 12^th-grade enrollments) will be included in the study. Half of the schools will be randomly selected into the treatment group, with the remaining placed in the control group (it will randomly be decided if the odd case is in the treatment or control group). This allows the study to have 80% power with an alpha of 0.05. The sampling design assumptions are shown in table 1.

The average class size of 44 masks the fact that there are only a few larger schools and many smaller schools; the median 12^th- or 11^th-grade school class is 19 students, and there are only about 15 schools with over 100 12^th-grade or 11^th-grade students. Stratification of schools will be by school size (below/above median enrollment) to ensure that large schools have an equal chance of being in either the treatment or control condition.

Table 1. Sample Design Assumptions

Assumption

α = 0.05

44 12^th graders and 44 11^th graders per school, on average

Power of 80%

Intraclass correlation of 0.15;

Proportion of variance explained by the REAs is 0.3

Proportion of variance explained by school covariates is 0.2

Estimated proportions of students being tested are 0.3 for the control group

Most schools in the study sample would be in their own school districts; i.e., assigning two or more schools within a district to different treatment and control conditions will likely not be possible. For example, ND contains a high number of school districts (148) relative to its number of schools with a 12^th grade (167), meaning that most school districts only contain one school with a 12^th grade. Though this theoretically limits the ability to reduce variation prior to assignment, in practice the student population of ND is homogeneous and the district policies and supports are fairly consistent. The primary diversity is between White and Native American students; White students make up about 83 percent of the ND student population and Native American students make up about 9 percent (Black and Hispanic students make up 2-3 percent of the population each; the remaining 2 percent are divided between Asian and Two or More Races). About 31 percent of the student population is free or reduced-price lunch eligible.

Nevertheless, schools will be assigned to different treatment and control conditions within a regional educational association (REA, a kind of super-district). There are eight REAs which provide support services to member school districts; two are small and will be combined into one, yielding seven REAs to use in stratifying the schools (with the size stratification being within REA). The REAs cooperate on programs and services to increase district operating efficiencies. Stratifying by REA enables us to split the schools in a REA between the treatment and control groups and can help ensure balance across experimental conditions.

In addition to the fact that there are few districts with multiple high schools, there are also few schools with multiple 12^th-grade or 11^th-grade classrooms, due to the small size of most ND high schools. Even where there are multiple classrooms or teachers in schools, the schools are still relatively small, which makes spillover effects likely. Thus classrooms within schools will not be assigned to different treatment and control conditions. Therefore, within each sampled school, all 11^th and 12^th graders will be included in the study.

In the western part of the state, where school growth (mostly at the elementary and middle school level) has been very rapid in recent years, there may also be a relatively greater number of transfer students; however, the numbers of 11^th- and 12^th-grade transfer students is expected to be low because workers in the gas industry are often not bringing their families with them initially—and would be less likely to do so if they have children nearing high school completion. Nevertheless, district-level differences in population are another aspect of the students’ environments about which information will be gathered.

B.3 Methods for Maximizing Response Rates and Deal with Nonresponse

Procedures for maximizing response rates at the institution and respondent levels are based on successful experiences in similar school-based studies. In this section, methods for maximizing response rates for schools, students, teachers, and school administrators are discussed.

The recruitment of schools into the study will begin with the dissemination of information to school counselors and district leaders to notify districts and schools about the study and to express the support of the state of North Dakota. School administrator conferences in late January/early February 2015, and a midwinter counselor’s conference in late February 2015 will be the vehicles for informing potential school participants of the study and its general outlines. The ND Department of CTE will announce the project in general terms and help build awareness of the importance of the study, as well as interest in participating, at these events. However, no information about selected schools or which schools are in the experimental or control group will be shared prior to the receipt of OMB approval.

After OMB clearance and IRB approval are received, we will contact districts and initiate the process of obtaining any required research applications for conducting research in their schools. Our contact with districts and schools will include cover letters that present the endorsement of the ND Department of CTE and the Department of Education, emphasizing the importance of the study and the value of its findings. All contacting materials are included in Appendix A.

Once any required district approval is obtained, we will begin contacting schools selected into the study through packets sent through Federal Express informing them of the nature of the study, its purpose, and its value to them and to the state, and asking for their participation. Packets will include endorsement letters, a sheet of frequently asked questions, and contact information to answer further questions. Several days after receipt of the package, a recruitment specialist from the NCICTE research team will contact the school, ask for their participation, address any questions or concerns they may have, and, if they agree to participate, begin the process of identifying a school staff member or members to serve as the RTS instructor and a separate (where possible) school coordinator to assist in the administration of the surveys and site visits. Recruitment specialists will document any refusals to participate and follow-up an additional time via email and phone to attempt to convert refusals into acceptance. The recruitment specialist will remain the primary point of contact between NCICTE and the school, coordinating site visits and ensuring continuity in communications and a single source of information and help for any issues that arise.

We will provide school coordinators with a website for the uploading and tracking of student survey participation and parental consent. North Dakota staff inform us that passive (implicit) parental consent is used statewide; therefore, consent forms will be sent to parents via students. Parents or guardians will receive a letter informing them of the nature of the study, the authorization to conduct it, the protection of their child’s data, and other information as required. They will also receive a consent form that they must return if they do not want their student to participate; such implicit consent usually yields higher participation rates than explicit consent. A sample cover letter and implicit consent form is also included in Appendix A. School coordinators will track parental consent forms and note exclusions for data collection (such students can receive the treatment but not be surveyed or otherwise have data gathered about them). School coordinators will also provide lists of names and other identifying information for students in the 11^th and 12^th grades, from which individual usernames and passwords will be generated, to enable the linking of student responses across the two surveys, and enabling eventual linking with administrative data. Since all students in the 11^th and 12^th grades will be included in the sample and survey participation will be a class activity, we anticipate a high level of student participation.

Plans for incentives are provided in Part A. Multiple incentives will be given to RTS instructors, school coordinators, and principals to motivate participation in the study. For teachers, counselors, or career advisors who will be delivering RTS, we will offer between $50-$100 for participation, in addition to paying for costs associated with training totaling no more than $200 per instructor (e.g., registration fees). Staff in ND will also work to enable instructors to receive college credit for participating in RTS training; this is believed to be a strong incentive for participation in the training (and therefore the study as a whole). This size of incentive is commensurate with the burden placed on instructors for completing two surveys throughout the school year for each class ($25 per survey). Principals at both treatment and control schools will receive a small incentive ($25) for their participation in principal interviews. School coordinators will receive up to a $150 incentive for managing consent forms, scheduling the student surveys, and coordinating with NCICTE researchers to plan site visit logistics where applicable. In addition, consistent with our experience on other interventions, we propose a school-level incentive for control schools which otherwise have little motivation to participate in the study.

B.4 Tests of Procedures or Methods to be Undertaken

As noted in the content justifications in Part A of this submission, items developed for the surveys drew upon existing surveys and RTS documentation to the greatest extent possible, and site visit protocols derive from prior experience conducting numerous similar case studies. Likewise, recruitment strategies and other communications materials were developed based on procedures utilized for NCES’s secondary longitudinal studies (High School Longitudinal Study of 2009; Education Longitudinal Study of 2002). Therefore, no tests of procedures or methods are planned for the study.

B.5 Individuals Consulted and Individuals Responsible for Collecting and Analyzing the Information

The following individuals in the U.S. Department of Education were consulted on the design, data collection, and analysis approach: Braden Goetz in the Office of Career, Technical, and Adult Education and Marsha Silverberg in the Institute of Education Sciences. The following RTI International staff work on the statistical and methodological aspects of the study design: Ben Dalton, Peter Siegel, Deborah Herget, and Michael Nelson. They can be reached at (919) 541-7228. The following FHI 360 staff work on methodological and analytical components pertaining to the site visits: Ivan Charner and Caitlin Rose Dailey.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Chapter 2
Author	spowell
File Modified	0000-00-00
File Created	2021-01-25