OMB83C Change Memo OFT & Recruitment for MS Base Year

MEMORANDUM OMB # 1850-0911 v.13

DATE: April 5, 2017

TO: Robert Sivinski and E. Ann Carson

Office of Information and Regulatory Affairs, Office of Management and Budget (OMB)

FROM: Carolyn Fidelman

National Center for Education Statistics (NCES)

THROUGH: Kashka Kubzdela

National Center for Education Statistics (NCES)

SUBJECT: Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) Recruitment for Main Study Base-year Change Request

The Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) is the first study conducted by the National Center for Education Statistics (NCES) to follow a nationally representative sample of students as they enter and move through the middle grades (grades 6-8). The data collected through repeated measures of key constructs will provide a rich descriptive picture of the academic experiences and development of students during these critical years and will allow researchers to examine associations between contextual factors and student outcomes. The study will focus on student achievement in mathematics and literacy along with measures of student socioemotional wellbeing and other outcomes. The study will also include students with disabilities for whom descriptive information on their outcomes, educational experiences, and special education services will be collected. Base-year data for MGLS:2017 will be collected from a nationally representative sample of sixth-grade students beginning in January 2018, with annual follow-ups beginning in January 2019 and in January 2020 when most of the students in the sample will be in grades 7 and 8, respectively. In preparation for the national data collection, referred to as the Main Study, the data collection instruments and procedures must be field tested. An Item Validation Field Test (IVFT) was conducted in the winter/spring 2016 to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be developed for the Main Study. An Operational Field Test (OFT) began in January 2017 to test the near-final instruments and the recruitment and data collection processes and procedures in preparation for the Main Study. District and school recruitment for the Main Study will begin in April 2017. OMB approved the Main Study recruitment and tracking of OFT students in December 2016 with the latest change request approved in January 2017 (OMB# 1850-0911 v.11-12).

This request is to (1) provide a recommendation for the Main Study school-level incentive based on the results from the IVFT and OFT school incentive experiments; (2) describe the school-level report we will offer schools as a benefit of participation; (3) describe general enhancements to the Main Study recruitment effort; (4) add study informational webinars as a mode of outreach to state, district, and school officials; (5) clarify that students will be able to keep the earbuds used during MGLS assessments; (6) recommend offering a food event as token of appreciation to OFT and MS students for returning explicit parental consent forms; (7) update the confidentiality pledge cited as part of MGLS:2017 to account for the Cybersecurity Enhancement Act of 2015 and cite data security and confidentiality requirements; and (8) present minor modifications in the language in the Main Study and OFT follow-up communication materials to be more compelling and likely to improve participation. These revisions do not affect the estimated burden to respondents or the total cost to the federal government for MGLS:2017 Main Study.

1. Main Study School-level Incentive

School participation has been increasingly difficult to secure. Given the many demands and outside pressures that schools already face, it is essential that schools see that MGLS:2017 staff understand the additional burden being placed on school staff when requesting their participation. The study asks for many kinds of information and cooperation from schools, including a student roster with basic demographic information (e.g., date of birth, sex, and race/ethnicity); information on students’ IEP status, math and special education teachers, and parent contact information; permission for field staff to be in the school for up to a week; space for administering student sessions (assessments and surveys); permission for students to leave their normal classes for the duration of the sessions; and information about the students’ teachers and parents. Sample students with disabilities may require accommodations and different session settings, such as individual administration and smaller group sessions, which will add to the time the study spends in schools and perhaps require additional assistance from school staff to assure that these students are accommodated appropriately.

The Operational Field Test (OFT) included a school-level incentive experiment, as described in the MGLS:2017 OFT and Recruitment for Main Study Base Year, Supporting Statement, Part A (OMB# 1850-0911 v.11). Schools were randomly assigned to one of three incentive conditions: Condition 1 - $200, Condition 2 - $400, or Condition 3 - $400 in materials or services for the school. (School coordinators also received $150, consistent across all three conditions.) The IVFT included the same three experimental conditions. Table 1 displays information on the types of non-monetary materials or services offered in Condition 3.

Table 1. Non-Monetary Incentive Choices for Schools in Experimental Condition 3

Incentive (Approximate Value = $400)

Registration for Association for Middle Level Education (AMLE) or Regional Annual Meeting

Two-Year School Membership in AMLE

Membership in Regional ML Organization plus Subscriptions to Professional Journals

Professional Development Webinar

School Supplies

Library of Middle Level Publications

The original analytic plans for the incentive experiment called for combining results for the IVFT and OFT schools, to increase the possibility of detecting differences with statistical significance. While we have provided an analysis that uses the combined results for the IVFT and OFT, we have also provided individual analyses for the IVFT and OFT given differences between those groups that affect their predictive value regarding the Main Study. The IVFT set of schools were purposively selected while the OFT schools were selected using a probability proportional to size sampling method that mimics the method to be employed for the main study. Also, the IVFT was fielded with a shortened school recruitment window, making it likely that recruitment results were affected. In addition, many districts refused to allow their sampled schools to be contacted for the study. This meant that sampled schools in those districts never actually received an incentive offer. To assess the degree to which the level of school incentive impacted participation rates, we analyzed results both including and excluding schools in districts that refused. Table 2 presents results for schools in all districts. Statistically significant differences between the incentive conditions were not found. Additionally, we found what appeared to be a notable difference in results between the IVFT and OFT.

Table 2. School Participation Rates by Experimental Condition (All Districts)

Experimental Condition		IVFT and OFT Combined Participation Rate (Number of Schools)	IVFT Participation Rate (Number of Schools)	OFT Participation Rate (Number of Schools)
1	$200	23.1% (30 of 130)	22.9% (20 of 87)	23.3% (10 of 43)
2	$400	27.3% (35 of 128)	22.1% (17 of 77)	35.3% (18 of 51)
3	$400 non-monetary equivalent	29.0% (36 of 124)	25.0% (21 of 84)	37.5% (15 of 40)

Table 3 presents participation rates among schools in cooperating districts. The differences between participation rates for the $200 condition versus the other two conditions in the IVFT, OFT, or IVT and OFT combined were not statistically different (p-value>.05).

Table 3. School Participation Rates by Experimental Condition (Participating Districts)

Experimental Condition		IVFT and OFT Combined Participation Rate (Number of Schools)	IVFT Participation Rate (Number of Schools)	OFT Participation Rate (Number of Schools)
1	$200	39.5% (30 of 76)	37.7% (20 of 53)	43.5% (10 of 23)
2	$400	49.3% (35 of 71)	37.8% (17 of 45)	69.2% (18 of 26)
3	$400 non-monetary equivalent	42.9% (36 of 84)	34.4% (21 of 61)	65.2% (15 of 23)

Table 4 presents participation rates for schools in all districts, when the two higher incentive level conditions are combined. The differences between participation rates for the $200 condition and the other two conditions for the IVFT, OFT, or IVT and OFT combined were not statistically different (p-value>.05).

Table 4. School Participation Rates by Combined Experimental Condition (All Districts)

Experimental Condition		IVFT and OFT Combined Participation Rate (Number of Schools)	IVFT Participation Rate (Number of Schools)	OFT Participation Rate (Number of Schools)
1	$200	23.1% (30 of 130)	22.9% (20 of 87)	23.3% (10 of 43)
2 and 3	$400 or $400 non-monetary equivalent	28.2% (71 of 252)	23.6% (38 of 161)	36.3% (33 of 91)

Table 5 presents participation rates among schools in cooperating districts, when the two higher incentive level conditions are combined. The differences between participation rates for the $200 condition and the other two conditions for the IVFT, OFT, or IVT and OFT combined were not statistically different; however, the participation rates under the OFT approached significance (p-value=.0542).

Table 5. School Participation Rates by Combined Experimental Condition (Participating Districts)

Experimental Condition		IVFT and OFT Combined Participation Rate (Number of Schools)	IVFT Participation Rate (Number of Schools)	OFT Participation Rate (Number of Schools)
1	$200	39.5% (30 of 76)	37.7% (20 of 53)	43.5% (10 of 23)
2 and 3	$400 or $400 non-monetary equivalent	45.8% (71 of 155)	35.9% (38 of 106)	67.4% (33 of 49)

We believe one explanation for the difference in results between the IVFT and the OFT may be the difference in respondent burden. The IVFT called for a much greater burden on schools, as we were sampling a larger number of students per grade, and sampling from grades 5 through 8; however, there was no request for longitudinal follow-up. The relatively small sample sizes for each field test also limited the ability to detect statistically significant differences. The OFT was a truer test of what we might expect in the Main Study, as we were sampling only from grade 6 and had a smaller student sample. With those greater similarities in mind, and a large difference in response rates in the OFT between treatment 1 versus 2 and 3, we recommend offering schools in the Main Study an incentive of $400, with the option of receiving it as a check or a non-monetary equivalent.

2. School-level Reports

An additional benefit to schools in participating is receiving information on how their school compares to others. We plan to provide each participating school a school-level report, which includes summary data on the aggregate performance of their students compared to the national average. We also propose including comparative data related to “schools like yours,” wherein data are presented for schools that are similar on key characteristics, such as urbanicity, sector (i.e., public/private), grade-levels served, geographic region, etc. We have produced such school-level reports for similar studies, such as the High School Longitudinal Study of 2009 (HSLS:09). The school characteristics selected have yet to be determined, as these reports will be produced in late 2019, after the base-year data file is released. Confidentiality of school identity and confidentiality/stability of student results will be maintained, meaning that “schools like yours” data and school-level aggregated-student data will not be included in reports when too few schools or too few students participated to ensure confidentiality and stability of estimates. The template for these reports will be provided in a change memo in 2019. We are seeking approval to offer the report to schools at this stage, as a means of thanking the schools for and encouraging their participation in the study. For this reason, school contacting materials have been revised to include mention of these school-level reports.

3. General Enhancements to Recruitment Activities

We developed a number of enhancements for the Main Study recruitment effort (noted in the Supporting Statement Part B) that are expected to encourage participation. These enhancements include increasing the in-school data collection window to allow schools more flexibility in scheduling test dates, conducting in-person recruitment visits to districts and schools to further explain the value of the study, and offering opportunities for data training after MGLS:2017 study data become available. In addition, we have established partnerships with additional organizations; including the National Center for Education, Research, and Technology (NCERT), and the School Superintendents Association (AASA). The study will use such partnerships to conduct outreach. Additional changes that may influence recruitment efforts are specified below (items 4-8).

4. Informational Webinars

We propose to host several informational webinars as part of our district/school recruitment efforts. The webinars will be designed to communicate the purpose and importance of the MGLS:2017, the value to schools, study milestones, and information on how schools can participate. Content has been drawn from OMB-approved communication materials included in the present package. Attendees would include state superintendents, state education research directors, district superintendents, district research directors, principals, school leaders, and others generally interested in middle grades issues. Attendees would remain anonymous. Representatives from both NCES and RTI will present. We will host the webinar twice in April, and whenever desirable until the end of the Main Study data collection period. Dates and times will be publicized via email to superintendent organizations, state and district representatives, posting on the NCES website, and social media. The webinar will be open to all who would like to attend, whether or not they have been sampled for the study. This will help smooth transitions should students transfer between schools during the study and reduce the potential risk associated with identification/association of selected schools/districts. We will archive the webinar for viewing at a convenient time for those who are unable to attend. The materials for the webinars are included as Appendix MS1-T.

5. Earbuds for Students Participating in Assessments/Questionnaires

Students will be using earbuds to complete the audio portion of the student assessment. Students will be allowed to keep the earbuds after participation, in addition to the already approved token incentives (e.g., keychain, pen). These tokens are mentioned in the parent consent materials.

6. Explicit Consent Form Return Food Event

Schools follow one of two models regarding parental consent. Schools with implicit consent notify parents about study activities and if parents object to their child’s participation, they need to return a signed form indicating their refusal. Schools with explicit consent notify parents every time there is an activity requiring parental permission and parents are asked to return a signed form indicating whether they do or do not consent to let the child participate in the activity. If parents do not return the form in explicit consent schools, the child is considered as not having consent to participate. It could be, however, that a parent would allow the child to participate and just forgot to return the form giving permission.

Results to date from the MGLS:2017 OFT have shown a consent rate of 89 percent in implicit consent schools and 52 percent in explicit consent schools. Approximately 30 percent of the OFT sampled schools have required explicit consent, making it a significant issue for the MGLS:2017 Main Study. The consent rate difference between these two sets of schools has the potential to introduce bias in the data. We believe this difference is likely attributable in part to parents forgetting to return the consent forms in explicit consent schools, as opposed to real, large differences in actual rates of refusal. We therefore propose adding a token of appreciation for students returning consent forms at explicit consent schools (regardless of whether consent is given or refused). Our expectation is that we will obtain consent for some students for whom we would have had to assume a refusal due to a non-returned form.

We propose to sponsor a food event, e.g., pizza, bagels, etc., for students who had returned completed parental consent forms by a set date. We recently observed a marked increase in return of forms at one of the explicit consent OFT schools where the school organized a pizza party to encourage the return of forms (25 percent of forms returned before announcement of pizza party, and 75 percent of forms returned at end of the week after the announcement). We would like to sponsor food events for the remaining OFT explicit consent schools and the explicit consent schools in the Main Study. If schools allow these food events, we will include mention of these events in the parent explicit consent materials.

7. Update to Confidentiality Pledge and Citation of Security and Confidentiality Requirements

Citation of the data security and confidentiality protection procedures has been expanded in Part A to reflect all of the laws and regulations with which MGLS:2017 data collection contractors and agents comply. Edits have been made throughout all Appendixes to ensure that the law citations are correctly referenced. Also, the confidentiality pledge has been updated throughout the submission documents (Part A, Appendixes MS1-C through MS1-T, Appendixes OFT2-A through OFT2-L, and note for Appendixes OFT1-T through OFT1-W) to reflect the addition of the Cybersecurity Enhancement Act of 2015 provision. The revised pledge reads: “All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).” The OFT base-year (OFT1-A through OFT1-S) materials and Main Study (MS1) Endorsement Request Letter and State Letter and Sample Endorsement Letter do not include the updated confidentiality pledge because they have already been used, given that OFT and MS endorsement and state letters became operational prior to the implementation of the revised pledge. All other aspects of the Main Study (MS1) and recruitment for OFT follow-up (OFT2) will include the revised pledge.

8. Main Study and OFT Follow-up Communication Materials

Minor edits to the district and school communication materials have been made to add emphasis on the importance of MGLS:2017 and the benefits of participating, as well as to expand the language related to confidentiality to include reference to FERPA. These edits apply to letters that have not yet gone out and, thus, are not included in OFT1-A through OFT1-S and MS1-A and MS1-B. The changes are listed below in Table 6 and reflected in the attached revised Appendices A-W documents.

Table 6. Proposed Edits to the MGLS:2017 Contacting Materials

Text Added	Text Deleted
Main Study FAQs
When is the study being conducted? MGLS:2017 will begin in early 2018 with students in sixth grade. Planned follow-ups with the same students will occur in the winter/spring of 2019 and the winter/spring of 2020.	When is the study being conducted? MGLS:2017 will begin in early 2018 with students in sixth grade. Planned follow-ups with the same students will occur in the spring of 2019 and 2020.
Why is MGLS:2017 important? MGLS:2017 will help educators, researchers, and policymakers learn about student growth in math and reading as well as family and school factors that encourage success during the middle grade years. The data collected will provide a rich, descriptive picture of the experiences and lives of young adolescents during this critical time and will permit researchers to examine associations between contextual factors and student outcomes. Because mathematics and literacy skills are important for preparing students for high school and are linked with later education and career opportunities, the study is placing a focus on instruction and student growth in these areas. Why participate in MGLS:2017? Over the past two decades, education policy has become more data driven with a recognition of the importance of reliable and systematically collected information at the federal, state, and local levels. MGLS:2017 will provide, for the first time, national data for a longitudinal cohort that can be used to examine various aspects and components of middle grades education, as well as evidence about the effectiveness of middle grades programs and practices.	Why participate in MGLS:2017? MGLS:2017 will help educators, researchers, and policymakers learn about growth in math and reading as well as family and school factors that encourage success during the important, yet generally understudied, middle grade years.
How will my school benefit from participating in MGLS:2017? The value to you of participating in MGLS:2017 includes extrinsic benefits, such as monetary incentives, but the true value is an intrinsic one. It is your opportunity as a dedicated supporter and advocate of the middle grades to positively contribute to the success of middle grades education for our current and future principals, teachers, students, and parents. The benefits of participation in MGLS:2017 include: Excitement—the feeling of being part of the first national longitudinal study of the middle grades by the U.S. Department of Education; Contribution—you can take pride in knowing that you are making a positive contribution to the field of middle grades education; and Understanding— if enough of your school’s students participate in the study, your school will receive a report showing how your students compare with students attending schools nationwide, as well as comparisons between your students and those at schools similar to yours on selected dimensions, such as public/private.	NA
Recruitment Website
Confidentiality: The Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) is conducted by the National Center for Education Statistics (NCES). Under contract with NCES, RTI International, a U.S.-based nonprofit research organization, administers MGLS:2017 on behalf of NCES. NCES is authorized to conduct MGLS:2017 by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543) and to collect students’ education records from education agencies and institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3) and 99.35). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). All project staff have signed confidentiality agreements and affidavits of nondisclosure and are prohibited by law from using the obtained information for any purposes other than this research study. You may review the Confidentiality Agreements and Affidavit of Nondisclosure statements (pdf, 81KB) that are signed by project staff. A nondisclosure form is also required before informing districts which schools are selected for the study. We take the confidentiality of participating schools seriously and limit disclosure only to those who need to know to make informed decisions about school and student participation in the study. A copy of the nondisclosure agreement may be accessed here. You may also review the procedures that govern NCES and RTI International to protect identifiable information.	Confidentiality: MGLS:2017 data are being collected for NCES by RTI International, a U.S.-based nonprofit research organization. The collected data may be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). The collected information will be combined across respondents to produce statistical reports. Students, parents, and schools are never identified in any reports. All RTI project staff have signed confidentiality agreements and affidavits of nondisclosure and are prohibited by law from using the obtained information for any purposes other than this research study. You may review the Confidentiality Agreements and Affidavit of Nondisclosure statements (pdf, 81KB) that are signed by project staff. A nondisclosure form is also required before informing districts which schools are selected for the study. We take the confidentiality of participating schools seriously and limit disclosure to those who need to know to make informed decisions about school and student participation in the study. A copy of the nondisclosure agreement may be accessed here. You may also review the procedures that govern NCES and RTI International to protect identifiable information.
Confidentiality and Paperwork Reduction Act Statements – Appendices A-W (beginning with MS1-C)
NCES is authorized to conduct MGLS:2017 by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3) and 99.35). The data are being collected for NCES by RTI International, a U.S.-based nonprofit research organization. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). The collected information will be combined across respondents to produce statistical reports. According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0911. Approval expires 09/30/2018. The time required to complete this information collection is estimated to average approximately <<time>> minutes per response, including the time to review instructions, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this survey, or any comments or concerns regarding the status of your individual submission of this survey, please write directly to: The Middle Grades Longitudinal Study of 2017-18 (MGLS:2017), National Center for Education Statistics, Potomac Center Plaza, 550 12th St, SW, Room 4002, Washington, DC 20202.	NCES is authorized to conduct MGLS:2017 by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). The data are being collected for NCES by RTI International, a U.S.-based nonprofit research organization. The collected data may be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law (ESRA 2002, 20 U.S.C., § 9573). The collected information will be combined across respondents to produce statistical reports. According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0911. Approval expires 09/30/2018. The time required to complete this information collection is estimated to average <<time>> minutes per response, including the time to review instructions, gather the data needed, and complete and review the information requested. If you have any comments concerning the accuracy of the time estimate, suggestions for improving the survey instrument, or concerns regarding the status of your individual response to this information collection, please write directly to:  Middle Grades Longitudinal Study (MGLS), National Center for Education Statistics (NCES), 550 12th St. SW, Room 4002, Washington, DC 20202.
Request for Student Roster Information from Schools
Not all students in your school are needed for participation. We will be selecting a sample. In order to select the sample, we are requesting a complete roster of the students currently enrolled in sixth grade at your school, including ungraded students whose primary IEP designations are one of the 3 focal IEP categories.	Not all students in your school are needed for participation. We will be selecting a sample. In order to select the sample, we are requesting a complete roster of the students currently enrolled in sixth grade at your school.
School Recruitment and Parent Permission Letters (Implicit Permission)
[As a thank you for participating, your child/Students] will be given a pair of earbuds used during the study and another small token of appreciation (e.g., keychain, pen).	N/A
Parent Permission Letter – Explicit Permission
As a thank you for participating, your child will be given a pair of earbuds used during the study and another small token of appreciation (e.g., keychain, pen). Students who return the attached form by [DATE] will be invited to a food event sponsored by the study.	N/A

4