Change Worksheet Memo

MEMORANDUM OMB # 1850-0911 v.5

DATE: September 3, 2015

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Carolyn Fidelman

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Middle Grades Longitudinal Study of 2017–18 (MGLS:2017) Recruitment for 2016 Item Validation Field Test Change Request

The Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) will be the first study sponsored by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education, to follow a nationally representative sample of students as they enter and move through the middle grades (grades 6-8). An Item Validation Field Test (IVFT) will be conducted to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be composed for the main study. An OMB package for the recruitment of schools, school districts, and parents to participate in the IVFT was approved on July 20, 2015 (OMB# 1850-0911 v.3) with the understanding that a change memo would be provided upon award of the MGLS contract based on revisions proposed by the contractor. The contract has been awarded to RTI and its partners.

This submission requests approval to (1) increase the school sample size; (2) to include a tier 2 sample, if needed, to achieve the necessary student response targets; (3) to adjust the burden table accordingly; (4) to simplify the collection of student roster information to a single step process; (5) to include outreach efforts by relevant organizations as part of the recruitment process; and (6) to update the recruitment materials. A separate generic clearance package will be submitted to conduct cognitive testing, using eye-tracking technology, of 30 students in each of the three focal disability groups as well as 30 students who are not identified in any disability group. As part of this request, the following text has been added to Part A:

Increased School Sample Size

The purpose of the IVFT is to test the instruments on at least 1,200 students in each of grades 6 through 8, 350 students in grade 5, and at least 200 respondents in each of three disability groups: specific learning disability, autism, and emotional disturbance. To achieve this goal, the number of participating schools in the IVFT should be at least 58 schools. A convenience school sample of about 250 schools will be selected for the IVFT from which to recruit the 58 schools. This not only assures the attainment of the requisite number of participating schools but also provides increased power to the previously proposed school incentive experiments. The larger school sample accounts for the challenge of securing school participation for the IVFT, given the brevity of the period between the start of recruitment and the start of IVFT data collection (September 2015 to January 2016). As originally proposed, schools will be randomly assigned to one of three incentive treatments: $200, $400, or $400 in materials or supplies.

The 250 schools selected for the IVFT will be recruited at the same time. A selection of the 58 schools representing a diversity among schools will be included in the IVFT. During this recruitment effort, participating schools will be tracked to ensure heterogeneity of demographics such as state and urbanicity. If, for example, a sufficient number of suburban schools from a particular state agree to participate, recruitment efforts with the remaining schools of that type will cease and focus will remain on the remaining schools in the state that are urban and rural. All schools who agree to participate will receive their assigned incentive regardless of their selection for participation. This will enable us to fully carry out the incentive experiment with all sampled schools regardless of their selection for participation. To provide context for the next section, schools in the sample for the IVFT are considered Tier 1 schools.

Tier 2 Schools

A study of this nature has not previously been undertaken and it is unknown whether 58 schools will be sufficient to attain the desired yield of students in each of the grades and disability groups. If it is determined that additional schools, beyond the 58, are needed to achieve the desired student yield within each of the subgroups, additional schools (Tier 2 schools) will be recruited to participate in only the student component of the study. For the purpose of the IVFT, collecting data from school staff and parents in the 58 participating sample schools should be sufficient to inform the operational field test and main study questionnaire testing. Thus, only students will be assessed in Tier 2 schools beyond the initial 58 participating schools to achieve the desired yield targets.

The opportunity for their students to participate in field-testing assessments for a national study is sometimes of considerable interest to school officials, and securing such “as needed” volunteer schools will safeguard the success of the IVFT. For these as-needed volunteer schools, depending on the school configurations and needs of the IVFT, participation may also be restricted to a subset of grades (e.g., one school may volunteer to have only their 5th-graders participate and another school may ask that only 8th-graders be included).

Tier 2 schools will be identified through a variety of means including the following activities:

• School officials (and district officials, if applicable) may provide positive response to volunteer participation requests made by middle-grades research and policy community organizations and representatives, including the Association for Middle Level Education (AMLE) and the National Forum to Accelerate Middle-Grades Reform (the Forum). There will be MGLS:2017 study representation (including the NCES project officer and RTI associate project director) and visibility (an exhibit booth and study update presentation) at the AMLE annual conference in October 2015 to provide information about the study, which may provide a mechanism for schools and districts to express their interest.

• Project personnel may identify volunteer schools through networking means, based on professional and personal relationships with various school- and district- officials.

• District officials that agree for their sampled schools to participate in the study may offer to have additional school(s) in their district included if needed.

• School officials at tier 2 schools may suggest additional schools that might be potential tier 2 volunteers.

Tier 2 schools will be considered an as-needed reserve pool of schools, and their participation in the IVFT will depend on the student yield overall and by various categories (e.g., grade level, disabilities oversample, school characteristics, and student characteristics).

Updated burden statement

The burden estimate (Exhibit 1) has been updated to reflect the additional schools sampled and additional participating schools. In addition, 10 minutes has been added to the original burden estimates for all districts and schools to debrief on reasons why schools or districts chose to participate or not to participate in the IVFT. The debrief may be completed via phone or email and will ask school administrators what factors led to the decision to participate or not to participate in MGLS:2017. For example, the debrief may ask whether content, incentives, time, and/or other factors impacted the decision. The information would only be collected from tier 1 schools and will be used to inform recruitment efforts for the OFT and the main study.

Table 5. Item Validation Field Test (IVFT) Recruitment Burden Estimates for Schools and Parents

Recruitment	Number of respondents and responses	Average burden time (minutes)	Total burden (hours)	Respondent average hourly wage1	Estimate of respondent labor cost
Nonparticipating districts	8	20	3	$44.13	$132
Participating districts	4	260	17	$44.13	$750
Nonparticipating schools	192	20	64	$44.13	$2,824
Participating schools	58	260	251	$44.13	$11,077
Tier 2 participating schools	20	180	60	$44.13	$2,648
Students’ parents	6,172	10	1,029	$22.71	$23,369
Total	6,454	-	1,424		$40,800

Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/datatype: Occupation codes: Parents - All employees (00-0000) and Education Administrators (11-9032); accessed on June 18, 2015.

As part of this request, the following text has been added to Part B:

Increased School Sample Size

A convenience school sample of approximately 92about 250 schools will need to be selected for the IVFT from which to recruit the 58 schools. The size of the school sample accounts for the challenge of securing school participation for the IVFT, given the brevity of the period between the start of recruitment and the start of IVFT data collection (September 2015 to January 2016).

We will purposively select 4 states and the District of Columbia, selecting states to represent each of the 4 Census geographic regions. In the 4 states, we will purposively select a single metropolitan statistical area (MSA). We will then select 50 schools from each state’s purposively selected MSA and select 50 schools from the DC MSA for a total of 250 schools; with the goal of obtaining 11-12 participating schools in each of the 4 states and the DC MSA. Some of the benefits of using MSAs over counties include:

(1) In states where districts and counties are synonymous, one doesn’t lose the entire area if a district refuses to participate;

(2) MSAs are better able to cover all of the urbanicity types; and, like counties,

(3) MSAs involve a reasonable travel range for field staff which leads to reduced collection costs due to the clustering of schools within defined geographic areas.

The 250 schools selected for the IVFT will be recruited at the same time. A selection of 58 schools from among those that agree to participate will be included in the IVFT. During this recruitment effort, participating schools will be tracked to ensure heterogeneity of demographics such as state and urbanicity. If, for example, a sufficient number of suburban schools from a particular state agree to participate, recruitment efforts with the remaining schools of that type will cease but will continue with the remaining schools in the state that are urban and rural.

Tier 2 Schools

If it is determined that additional schools, beyond the 58 Tier 1 schools, are needed to achieve the desired student yield within each of the subgroups, additional schools (Tier 2 schools) will be recruited to participate in the student component of the study. For the purpose of the IVFT, collecting data from school staff and parents in the 58 Tier 1 participating sample schools should be sufficient to inform the operational field test and main study questionnaire testing. Thus, only students will be assessed in Tier 2 schools to achieve the desired yield targets.

The opportunity for their students to participate in field-testing assessments for a national study is sometimes of considerable interest to school officials, and securing such “as needed” volunteer schools will safeguard the success of the IVFT. For these as-needed volunteer schools, depending on the school configurations and needs of the IVFT, participation may also be restricted to a subset of grades (e.g., one school may volunteer to have only their 5th-graders participate and another school may ask that only 8th-graders be included).

Simplifying Student Roster Collection

Once a school or district has agreed to participate, the MGLS contractor, RTI, will gather student rosters from the district or with the assistance of the school coordinator. Gathering information on the students will be a two-step process. First, a from the school. A complete roster of all students eligible for sampling will be requested, including key student characteristics, such as: name; ID number; date of birth; grade level; gender; race/ethnicity; and IEP status with disability code(s), when applicable. Based on this information the student sample will be drawn. Next, to completeAs part of the roster information, within 3-5 business days, the MGLS contractorcollection, RTI will also request from the school coordinator or designated district personnel the following information for each student selected into the sampleeligible for sampling: student’s parent and/or guardian contact information (e.g., mailing address; landline phone number; cell phone number; e-mail address); student’s math teacher; and student’s special education teacher, when applicable. Schools and districts usually find it easiest, and therefore most efficient, to supply all of the desired information one time for all of their students. However, should it be problematic for any school or district to provide the parent and teacher information on the complete roster, RTI will gather that information as a second step for the sampled students only.

The roster form in Appendix T has been revised accordingly.

Outreach

As part of the strategy to maximize response rates among school districts and schools during the recruitment process, RTI has established partnerships with organizations such as the Association for Middle Level Education (AMLE) and the National Forum to Accelerate Middle-Grades Reform (the Forum). These organizations will actively promote the value of the study to their constituencies as will a number of middle-grades education researchers who will participate in the recruitment effort.

Representatives from these organizations have committed to provide outreach to sample schools via direct letters and emails, and to the middle grades community in general via information in newsletters and related communications. These communications will include information about the importance of the study, what study participation entails, and the benefits of the study to the middle grades community. Direct communications with the schools selected for the IVFT will let schools know that they may also be selected for the main study (the IVFT will take place during the 2015-16 school year, while the main study will take place two years later, during the 2017-18 school year).

As part of this request, text has been revised in Appendices A-S:

Contact materials

The IVFT endorser, state, district, school, teacher, and parent recruitment materials (Appendices A-S) have been updated, and mentioned to NCES using Facebook or Twitter account to promote the study in the IVFT have been removed.