Part A MGLS 2017 IV Field Test 2016 Recruitment

The Middle Grades Longitudinal Study of 2017-2018 (MGLS:2017) will be the first study sponsored by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education, to follow a nationally representative sample of students as they enter and move through the middle grades (grades 6–8). In preparation for the main study, the data collection instruments and procedures must be field tested. This submission describes the overarching plan for the recruitment of schools, school districts, and parents to participate in the MGLS:2017 Item Validation Field Test (IVFT). A separate OMB clearance request for the field test data collection will be submitted in September 2015.

The primary purpose of the IVFT is to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be composed for the main study. Part A of this submission presents information on the basic design of the IVFT; Part B on the collection of information employing statistical methods; and Appendices A through T provide field test recruitment and student roster collection materials, consisting of letters to state and district officials, school principals, and parents, as well as text for an MGLS:2017 brochure, frequently asked questions, and website. Additionally, as some schools and/or districts may require a submission of a research application that requests detailed information on the proposed assessments and surveys, Appendices K through R provide content summaries of the proposed assessments and surveys, and Appendix U provides the current drafts of the survey instruments. The MGLS:2017 contract covering the IVFT through the main study’s 7th-grade data collection was recently awarded to RTI International (a trade name of the Research Triangle Institute). This document contains revisions described in the change memo to the document originally approved in July 2015 (OMB# 1850-0911 v.3).

A. Justification

A.1 Importance of Information

The MGLS:2017 will be the first study sponsored by NCES to follow a nationally representative sample of students as they enter and move through the middle grades (grades 6–8). A study of the middle grades will complement NCES’s plans for implementing a multi-cohort sequence for its longitudinal studies series. This means that the Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011), the MGLS:2017, and the High School Longitudinal Study of 2020 (HSLS:2020) will work toward synchronizing within a given 10-year span to collect the full range of data on students’ school experiences as they transition from elementary school into high school. The federal government is uniquely positioned to undertake the needed comprehensive large-scale longitudinal study of a nationally representative sample of middle-grade youth that includes measures of known critical influences on adolescents’ academic and socioemotional trajectories. NCES is authorized to conduct the MGLS:2017 under the Education Sciences Reform Act of 2002 (20 U.S. Code, Section 9543).

The MGLS:2017 will be a nationally representative sample of students enrolled in sixth grade during the 2017–18 school year, with the baseline data collection taking place from January through March of 2018, possibly extending through June 2018. Annual follow-ups are planned for winters of the 2018-2019 and 2019-2020 school years, when most of the students in the sample will be in grades 7 and 8, respectively. The MGLS:2017 will provide a rich descriptive picture of the academic experiences and development of students during these critical years and will allow researchers to examine associations between contextual factors and student outcomes. There is a wealth of research highlighting the importance of mathematics and literacy skills for success in high school and subsequent associations with later education and career opportunities. Thus, the study will focus on student achievement in these areas, along with measures of student socioemotional well-being and other outcomes. The study will also include a sample of students with different types of disabilities that will provide descriptive information on their outcomes, educational experiences, and special education services.

The MGLS:2017 will rely on a set of longitudinal and complementary instruments to collect data across several types of respondents to provide information on the outcomes, experiences, and perspectives of students across grades 6, 7, and 8; their families and home lives; their teachers, classrooms, and instruction; and the school settings, programs, and services available to them. At each wave of data collection in the main study, students’ mathematics and reading skills, socioemotional development, and executive function will be assessed. Students will also complete a survey that asks about their engagement in school, out-of-school experiences, peer group relationships, and identity development. Parents will be asked about their background, family resources, and involvement with their child’s education and their school. Students’ mathematics teachers will complete a two-part survey: In part 1, they will be asked about their background and classroom instruction. In part 2, they will be asked to report on the academic behavior, mathematics performance, and classroom conduct of each study child in their classroom. For students receiving special education services, their special education teacher or provider will also complete a survey questionnaire similar in structure to the two-part mathematics teacher instrument, consisting of a teacher-level questionnaire and student-level questionnaire, but with questions specific to the special education experiences and services of the study child. School administrators will be asked to report on school programs and services, as well as on school climate.

In short, the MGLS:2017 aims to provide data on the development and learning that occur during students’ middle-grade years (grades 6–8) and that are predictive of future success, along with the individual, social, and contextual factors that are related to successful development. A key goal of the study is to provide researchers and policymakers with the information they need to better understand the school and nonschool influences associated with mathematics and reading success, socioemotional health, and positive life development during the middle-grade years and beyond. To support the development of the study, the MGLS:2017 is conducting two field tests, the IVFT beginning in January 2016, followed by the Operational Field Test (OFT) that will begin in January 2017.

The study’s success is dependent on the development of reliable, valid measures. The goal of the IVFT is to collect data to support examination of the mathematics assessment, reading assessment, executive function assessment, student survey, parent survey, and school staff surveys. The IVFT will provide the data needed to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be composed for the main study. As the focus of the IVFT is the analyses of the psychometric properties of the survey items and assessments, the IVFT requires a large, diverse field test sample, though not a nationally representative one.

Gaining schools’ cooperation in voluntary research is increasingly challenging. The OFT will be used to test materials and procedures revised based on the results of the IVFT and to gain a deeper understanding of effective recruitment strategies that lead to higher response rates and thus better data quality. The OFT will include a responsive design approach for non-responders and will allow NCES to tighten assessment and survey timing, so as to maximize the overall functionality of the assessments and surveys while minimizing the time it takes respondents to complete them. The OFT results will inform modifications to the main study materials and procedures. With the focus of the OFT on recruitment strategies, tactics for retention of the sample within the study, and the operational administration of the surveys and assessments, the OFT will require a close to nationally representative sample.

A.2 Purposes and Uses of Data

The IVFT will take place in January through June 2016. Its purpose is to evaluate a battery of student assessments (i.e., mathematics and reading skills and executive function) and survey instruments (i.e., student survey, parent survey, and school staff surveys) for use in the MGLS:2017 Operational Field Test and in turn the MGLS:2017 main study collections. The IVFT will collect data from children enrolled in 5^th through 8^th grade as of January of 2016 and will provide the much needed information to establish the validity and reliability of the direct assessments and surveys.

Field Test Components

The IVFT includes the following components: student assessments and student survey, parent survey, math teacher survey, special education teacher survey, and school administrator survey.

Student Assessments and Student Survey. Students will participate in assessments and a survey, designed to take approximately 90 minutes per student.

Mathematics Assessment. The MGLS:2017 main study mathematics assessment will be a 30-minute, two-stage adaptive assessment that students will take on a tablet computer. The focus will be on domains of mathematics that are most likely to be the central focus of middle school learning now and in the future: the Number System, Ratios and Proportional Relationships, Expressions and Equations, and Functions. To ensure that we are sensitive to the variation in students’ mathematics ability, the assessment will include items with appropriately varying cognitive demand. The MGLS:2017 mathematics assessment will provide valuable information about the development of middle-grade students’ knowledge of mathematics and their ability to use that knowledge to solve problems, moving toward stronger reasoning and understanding of more advanced mathematics.
Reading Assessment. The MGLS:2017 reading assessment will use a two-stage adaptive assessment design consisting of a brief routing block (first stage: approximately 10 minutes) followed by a skill-based block (second stage: approximately 20 minutes), for a total of 30 minutes. The routing block will include items that measure foundational components of reading that are important for comprehension: Vocabulary, Morphological Awareness, and Sentence Processing. Performance on the routing block will direct students to one of three types of skill-based reading blocks (basic components, basic comprehension, or scenario-based comprehension) within the second stage.

The second-stage basic components skill block will be used to gather more information on the foundational reading component skills, including those measured in the first stage as well as word recognition and decoding skills. The basic components block will also capture information about students’ efficiency at basic reading comprehension and ability to comprehend short passages. The second-stage basic comprehension skill block is designed to gather information about students’ efficiency at basic reading comprehension and their ability to comprehend short passages. This skill-based block will measure comprehension in a traditional design where unrelated passages and corresponding questions are presented. The second-stage scenario-based comprehension skill block is designed to gather information about students’ ability to comprehend informational text and reason more deeply about text and to apply what they learn from passages. The scenario-based block will include a scenario or a purpose for reading (e.g., preparing for a classroom discussion or creating a website on a topic).

Executive Function Measures. Executive function, a set of capacities and processes originating in the prefrontal cortex of the brain, permits individuals to self-regulate, engage in purposeful and goal-directed behaviors, and conduct themselves in a socially appropriate manner. Self-regulation is needed for social success, academic and career success, and good health outcomes. Executive function includes capacities such as shifting (cognitive and attention flexibility), inhibitory control, and working memory. Four different executive function measures will be included in the field tests: Stop Signal (inhibitory control), 3-Back with verbal stimulus (working memory), 2-Back with nonverbal stimulus (working memory), and the Hearts and Flowers task (shifting or cognitive flexibility).
Student Survey. The purpose of the student survey is to collect information on students’ attitudes and behaviors; out-of-school time use; and family, school, and classroom environments. The student survey will also serve as a source for information about socioemotional outcomes having to do with social relationships, support, and school engagement.

Parent Survey. The parent survey will take 30 minutes to complete via a self-administered web-based questionnaire; a telephone interview follow-up will be available for respondents who do not complete the questionnaire via the web. The parent survey will focus on supplementing the information collected from students and teachers about the students’ educational experiences and on learning about parents’ expectations for their children’s academic attainment in high school and beyond. It will also collect information about family involvement in the children’s education and about family characteristics that are key predictors of academic achievement and other student outcomes.

Mathematics Teacher Survey/Teacher Student Report. The mathematics teacher survey will consist of two parts: a teacher survey and a series of teacher student reports (TSRs). Both the mathematics teacher survey and the TSR will be web-based, self-administered surveys, with a phone interview option available. The mathematics teacher survey is expected to take approximately 20 minutes to complete, and the TSR will take 5 to 10 minutes for each student that is rated. The mathematics teacher survey will collect data on potential classroom-level correlates of students’ mathematics achievement as well as school-level services and factors such as special programs, school climate, and instructional leadership.

Teacher responses to the TSR will capture information specific to the sampled student and his or her mathematics class. It will provide information on the classroom attendance and performance of individual students, which will augment direct student assessments, transcript information, and student and parent reports. The TSR will also serve as an additional source for data on student socioemotional outcomes related to regulation, school engagement, and externalizing behaviors. In the web version of this instrument, teachers will be given a list of the students for whom they should complete a TSR and will click on each student’s name to launch the TSR for that specific student. If a teacher opts not to complete the web-based survey, a follow-up phone interview will be conducted.

Special Education Teacher Survey/Teacher Student Report. Like the mathematics teacher survey, the special education teacher/service provider survey will consist of two parts. The first part consists of the teacher questionnaire, which asks questions about the teacher’s background and experiences working with students with disabilities. The second part contains the TSR, which contains specific questions about special education services and other contextual variables for sampled children with an Individualized Education Program (IEP) as well as ratings of individual academic and life skills (the special educator rating scale, SPERS).

The special education teacher survey will be web based and self-administered, with a phone interview option available. The first part of the survey will take approximately 10 minutes to complete, and the second part will take about 25 minutes for each student. In the second part of the web version of this instrument, teachers will be given a list of the students for whom they should complete the survey and will click on each student’s name to launch this part for that specific student.

School Administrator Survey. The school administrator survey will be web based and self-administered, with a telephone option available, and will take the administrator (generally, the principal or principal’s designee) approximately 20 minutes to complete. The school administrator survey will collect information about a school’s characteristics and staffing (specifically, the school’s structure and climate, including safety, organization, and support). It will also collect information on the student population, student conduct, academic culture, course offerings, and extended learning opportunities (e.g., extracurricular activities, summer school, or supports for struggling students).

Administration of Assessments and Survey Components

In the IVFT, students’ parents, math teachers, special education providers (as applicable), and school administrators will be asked to complete surveys as described above. To keep student participation to approximately 90 minutes and gain as much information on as many assessment and survey items as possible, the IVFT will employ a spiral design in which not all students will receive the same assessments and survey items. Table 1 below presents a summary of the student assessment and survey booklet spiral design.

Table 1. Item Validation Field Test Student Assessment and Spiral Design

Booklet 1	Booklet 2	Booklet 3	Booklet 4	Booklet 5	Booklet 6
Math assessment	Math assessment	Math assessment	Math assessment	Math assessment	Math assessment
Demographic items	Executive function task: Hearts & Flowers	Executive function task: 3-Back	Demographic items	Demographic items	Demographic items
Reading assessment (two-stage)	No reading assessment	No reading Assessment	Reading assessment (two-stage)	Reading assessment (two-stage)	Reading assessment (two-stage)
Executive function task: Stop Signal	Demographic items	Demographic items	Executive function task: 2-Back	No executive function task	No executive function task
Theories of Intelligence (general)	Student Questionnaire	Student Questionnaire	No student questionnaire	No student questionnaire	No student questionnaire

School Recruitment Approach

The student sample for the IVFT, while not required to be nationally representative for psychometric analysis, will include students in the typical age range found in grades 5–8 in the United States; these students will likely demonstrate a range of ability on the constructs being measured by the MGLS item pool. The sample will also include a subset of students from the three focal disability groups (learning disability, autism, and emotional disturbance) who are able to take standardized tests using accommodations. Schools will be recruited both directly and at the district level.

A.3 Improved Information Technology (Reduction of Burden)

Where feasible, available technology will be used to reduce burden and improve efficiency and accuracy. For example, if districts can provide information linking students to their mathematics teachers or students with disabilities to their special education teachers electronically, we will use this information rather than asking for it at the school level. The burden of recruitment on districts and schools will be minimal, with most information gathered over the telephone. Districts will primarily be asked to provide confirmation of data gathered from other sources, including school universe files and district and school websites. Our collection of student lists will accommodate whatever format districts and schools find to be the least burdensome. The study will utilize the information in any format it is provided.

A.4 Efforts to Identify Duplication

The MGLS:2017 will not be duplicative of other studies. While NCES longitudinal studies have contributed to our understanding of the factors that influence student success and failure in school, the middle grades (grades 6–8) are noticeably absent from the studies conducted to date. A majority of nationally representative longitudinal studies have focused on high school students and on the transition from secondary to postsecondary education: e.g., the High School and Beyond Longitudinal Study (HS&B) and the Education Longitudinal Study of 2002 (ELS:2002). The Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), and the National Education Longitudinal Study of 1988 (NELS:88) collected data on students in grade 8, but neither included a data collection in grades 6 and 7. The ECLS-K:2011 will not follow students beyond grade 5, and the High School Longitudinal Study of 2009 (HSLS:09) began with a national sample of students in grade 9. Thus, there is little information at the national level about the learning that occurs during grades 6–8 and about the rates of learning for different groups of students who may experience diverse school environments and opportunities.

The MGLS:2017 is unique in that it will assess students’ mathematics and reading achievement, as well as other student outcomes (e.g., executive function and socioemotional development), for the same group of students over a 3-year period. In addition to the ECLS-K and NELS:88, other national studies have assessed some of these outcomes for students in grade 8, including the National Assessment of Educational Progress (NAEP) and the Trends in International Mathematics and Science Study (TIMSS). These studies, however, are cross-sectional and do not include repeated measures of achievement or assess multiple subjects and areas of development for the same sample of students. Therefore, they cannot answer questions about students’ growth in mathematics and reading over the middle-grade years, about differences in the rates of growth for different populations (e.g., differences by gender, by race/ethnicity, and for students attending public and private schools), and about the school and nonschool factors that may facilitate or hinder this growth. Nor can they explore questions about the relationships between student achievement and other school outcomes and executive functions (e.g., working memory, attention, and inhibitory control) that work to regulate and orchestrate cognition, emotion, and behavior to enable a student to learn in the classroom. The MGLS:2017 will also be unique in its focus on obtaining a sample of students in three disability categories that can be studied on their own or compared to general education students over the three middle level years.

Other adolescent development studies have been conducted, but they often do not include a grade 6 sample. For example, the youngest children in the National Longitudinal Study of Adolescent Health (Add Health) and the Maryland Adolescent Development in Context Study (MADICS) were in grade 7 at baseline. Many of these studies collected data on local samples, had a primary focus on family and child processes, and were started in the 1990s: e.g., MADICS and the Michigan Study of Adolescent and Adult Life Transitions (MSALT). As such, they do not provide a contemporary picture of U.S. students in grades 6–8.

A.5 Minimizing Burden for Small Entities

Although small entities are not part of this study, in general, burden will be minimized wherever possible. During district and school recruitment, we will minimize burden by training recruitment staff to make their contacts as straightforward and concise as possible. The recruitment letters and materials (e.g., the study description and FAQs) are designed to be clear, brief, and informative. In addition, contractor staff will conduct all test administration and will assist with parental notification, sampling, and other study tasks as much as possible within each school.

A.6 Frequency of Data Collection

The MGLS:2017 IVFT is a one-time data collection that will take place in January through June 2016.

A.7 Special Circumstances

There are no special circumstances involved with the recruitment.

A.8 Consultations Outside NCES

As part of the MGLS:2017 design contract, content experts were consulted in the development of the assessments and questionnaires. These experts are listed by name, affiliation, and expertise in table 2.

Table 2. Members of the MGLS:2017 Content Review Panels

Name	Affiliation	Expertise
Mathematics Assessment Content Review Panel (June 18–19, 2013)
Tom Loveless	Brookings Institution	Policy, math curriculum
Linda Wilson	Formerly with Project 2061	Math education, math assessment, middle school assessment, author of NCTM Assessment Standards for School Mathematics and NAEP math framework, teacher
Kathleen Heid	University of Florida	Math education, use of technology, teacher knowledge, NAEP Grade 8 Mathematics Standing Committee member
Edward Nolan	Montgomery County Schools, Maryland	Math curriculum and standards, large-scale assessment of middle-grade students
Lisa Keller	University of Massachusetts, Amherst	Psychometrics, former math teacher
Paul Sally	University of Chicago	Math education, mathematics reasoning, mathematically talented adolescents
Margie Hill	University of Kansas	Co-author of Kansas mathematics standards, former NAEP Mathematics Standing Committee member, former district math supervisor
Executive Function Content Review Panel (July 18, 2013)
Lisa Jacobson	Johns Hopkins University; Kennedy Krieger Institute	Development of executive functioning skills, attention, neurodevelopmental disorders, and parent and teacher scaffolding
Dan Romer	University of Pennsylvania	Adolescent risk taking
James Byrnes	Temple University	Self-regulation, decision making, cognitive processes in mathematics learning
Socioemotional-Student-Family Content Review Panel (July 25–26, 2013)
James Byrnes	Temple University	Self-regulation, decision making, cognitive processes in mathematics learning
Russell Rumberger	University of California, Santa Barbara	School dropouts, ethnic and language minority student achievement
Tama Leventhal	Tufts University	Family context, adolescence, social policy, community and neighborhood indicators
Susan Dauber	Bluestocking Research	School organization, educational transitions, urban education, parent involvement and family processes
Scott Gest	Pennsylvania State University	Social networking, social skills, longitudinal assessment of at-risk populations
Kathryn Wentzel	University of Maryland	Social and academic motivation, self-regulation, school adjustment, peer relationships, teacher-student relationships, family-school linkages
Richard Lerner	Tufts University	Adolescent development and relationships with peers, families, schools, and communities
School Administrator Content Review Panel (August 16, 2013)
Susan Dauber	Bluestocking Research	School organization, educational transitions, urban education, parent involvement and family processes
George Farkas	University of California, Irvine	Schooling equity and human resources
Jeremy Finn	State University of New York at Buffalo	School organization, school dropouts
Edward Nolan	Montgomery County Schools, Maryland	Large urban school system administrator
Tom Loveless	Brookings Institution	Policy, math curriculum
Reading Assessment Content Review Panel ( April 14, 2014)
Donna Alvermann	University of Georgia	Adolescent literacy, online literacy, codirector of the National Reading Research Center (funded by the U.S. Department of Education)
Joseph Magliano	Northern Illinois University	Cognitive processes that support comprehension, the nature of memory representations for events depicted in text and film, strategies to detect and help struggling readers
Sheryl Lazarus	University of Minnesota	Education policy issues related to the inclusion of students with disabilities in assessments used for accountability purposes, student participation and accommodations, alternate assessments, technology-enhanced assessments, teacher effectiveness, large-scale assessments, school accountability, research design (including cost analyses), data-driven decision making, rural education, the economics of education
Disabilities Content Review Panel (April 29, 2014)
Jose Blackorby	SRI International	Autism, specific learning disabilities, special education, curriculum design, alternate student assessment, large-scale studies of students with disabilities, codirector of the Special Education Elementary Longitudinal Study (SEELS)
Lynn Fuchs	Vanderbilt University	Specific learning disabilities, student assessment, mathematics curriculum, psychometric models
Mitchell L. Yell	University of South Carolina	Autism, emotional and behavior disorders, specific learning disabilities, pre-K–12 instruction and curriculum, special education, evidence-based intervention
Sheryl Lazarus	University of Minnesota	Special education policy, inclusion of students with disabilities in assessments, accommodations, alternate assessments, technology-enhanced assessments, large-scale assessments, school accountability, research design (including cost analyses)
Martha Thurlow	University of Minnesota	Specific learning disabilities, reading assessment, alternate student assessment, early childhood education, special education, curriculum, large-scale studies
Diane Pedrotty Bryant	University of Texas, Austin	Educational interventions for improving the mathematics and reading performance of students with learning disabilities, the use of assistive technology for individuals with disabilities, interventions for students with learning disabilities and who are at risk for educational difficulties

A.9 Payments or Gifts to Respondents

High levels of school participation are critical to the success of the IVFT. School administrator, mathematics teacher, special education teacher, parent, and student data collection activities are contingent on school cooperation. NCES recognizes that the burden level of the study is one of the factors that school administrators will consider when deciding whether to participate. To offset the perceived burden of participation, NCES intends to continue to use strategies that have worked successfully in other major NCES studies (e.g., ECLS-K, ECLS-K:2011, HS&B, NELS:88, and ELS:2002), including offering both monetary and non-monetary incentives. Table 3 summarizes the proposed incentive amount for each instrument and activity along with their estimated administration times; a brief justification for each incentive amount follows table 3.

Table 3. Item Validation Field Test (IVFT) Instruments and Proposed Incentive Amounts

Instrument/Activity	Administration Time*	Field Test Incentives
Student Assessments and Survey (Math, Reading, Executive Function, and Student Survey)	90 minutes	No monetary incentive
Parent Survey	30 minutes	None, $20, or $40
Mathematics Teacher
Teacher Survey	20 minutes	$20
Teacher Student Report	10 minutes per student	$7 per TSR
Special Education Teacher
Teacher Survey	10 minutes	$20
Teacher Student Report	25 minutes per student	$7 per TSR
School Administrator Survey	20 minutes	No monetary incentive
School Participation School Coordinator (logistics, on-site visit, consent forms, administrative records, etc.)	6 hours for consent assistance 2 hours to schedule assessments 2 hours to set up web access, coordinate computer labs 6 hours to provide administrative records	$200, $400, or $400 in material or services for school $150 for coordinator

*Note that the assessment administration time may be longer for students with disabilities.

Students

There is no monetary incentive in the IVFT for students.

Parents

Parent survey response rates have declined over the past decade. The ECLS-K:2011 baseline (fall 2010) parent survey response rate was more than 10 percentage points lower (74 percent)¹ than the parent survey rate in the corresponding 1998 wave of the ECLS-K (85 percent).² Additionally, the 9^th grade parent survey response rate for the HSLS:09 baseline was 68 percent.³ The MGLS parent survey is a key component of the data being collected.

To improve the chances of obtaining higher parent participation rates in a school-based design, we will work with school personnel to recruit sample students’ parents into the MGLS:2017 and will conduct an experiment in the IVFT to determine the effect of different levels of monetary incentives on parent participation.

In the IVFT, an experiment will be used to determine the effect on response rates and on the cost and length of nonresponse follow-up of offering parents of middle-grade students a $0, $20, or $40 incentive for completing the parent questionnaire. Additionally, the experiment will also evaluate whether parents of children with disabilities, who may be more reluctant to engage in this study and who may require more frequent and extensive nonresponse follow-up, are influenced differently by the offer of a monetary incentive than parents of students without disabilities.

For parent incentives each school will be randomly assigned to one of the three experimental conditions. Therefore, all parents asked to complete the parent survey within a school will be assigned to the same condition. Parents in one-third of the schools will be asked to complete the parent survey, but will not be offered a monetary incentive for doing so; parents in another one-third of the schools will be offered $20 to complete the survey; and parents in the remaining one-third of the schools will be offered $40 to complete the survey. We will monitor the response rate in each group and document the level of effort needed to obtain the response rates achieved under the different incentive/no-incentive options. All groups will receive similar reminders and other modes of follow-up contact. The number of contact attempts to achieve the final response rates will be measured to compute the potential resource savings, if any, of each incentive payment relative to no incentive payment.

For the IVFT, as shown in Part B section B.1 we plan for 3,950 participating students. Assuming an 80 percent response rate from students, this means we will have needed to consent 4,938 students. Therefore, within the IVFT, we will be seeking parent surveys from 4,938 students’ parents. Assuming 4,938 are split approximately equally across schools and conditions; this would result in approximately 1,646 cases within each incentive level.

As stated, the 9^th grade parent survey response rate for the HSLS:09 baseline was 68%. For a power of 0.80, a confidence level of 95%, and 1,646 cases within each condition, in this experiment a 4.5% point difference in response rate should be detectable as statistically significant (e.g., 68.0% vs. 72.5%). Formula provided below.^⁴

n = (Z_α/2+Z_β)² * (p₁(1-p₁)+p₂(1-p₂)) / (p₁-p₂)²

Where Z_α/2 is the critical value of the Normal distribution at α/2 (e.g. for a confidence level of 95%, α is 0.05 and the critical value is 1.96); Z_β is the critical value of the Normal distribution at β (e.g. for a power of 80%, β is 0.2 and the critical value is 0.84) and p₁ and p₂ are the expected sample proportions of the two groups.

However, the IVFT has a clustered design with students nested in schools. Therefore, assuming an approximate design effect of 4, which is a similar design effect as reported by the HSLS:09 for parent respondents^⁵, which also had a clustered design with students nested in schools, the effective sample size for any condition would be approximately 412 cases (1,646/4). For a power of 0.80, a confidence level of 95%, and 412 cases within each condition, this experiment should be able to detect approximately an 8.5% point difference in response as statistically significant (e.g., 68.0% vs. 76.5%).

Teachers

The incentive proposed for students’ teachers is $20 per teacher survey, plus $7 per teacher student report (TSR). These amounts are consistent with the amounts used in current NCES studies, such as the ECLS-K:2011. While it is estimated that the mathematics teacher survey will take longer to complete (20 minutes) than the special education teacher survey (10 minutes), the reverse is true for the individual student reports. The individual student reports will require approximately 10 minutes per student to complete for mathematics teachers and 25 minutes per student to complete for special education teachers (including 5 minutes for an indirect assessment of student’s skills, the SPERS).

Schools and School Coordinators

As part of the IVFT schools recruitment, we propose to conduct an incentive experiment. Each school will be randomly assigned to one of the three experimental conditions. Given the many demands and outside pressures that schools already face, it is essential that they see that MGLS:2017 staff understand the additional burden being placed on school staff when requesting their participation. The study asks for many kinds of information and cooperation from schools, including a student roster with basic demographic information (e.g., date of birth, sex, and race/ethnicity); information on students’ IEP status, math and special education teachers, and parent contact information; permission for field staff to be in the school for up to a week; space for administering student assessments; permission for students to leave their normal classes for the duration of the assessments; and information about the students’ teachers and parents. For sample students with disabilities, on average, 5 students in each school will be selected based on disability category, and many will require accommodations and different assessment settings, such as individual administration and smaller group sessions. Working with the data collection contractor to assess these students will place even more of a burden on the participating schools.

In Condition 1, the baseline condition, we will offer schools a $200 incentive for participation. This amount is consistent with the amount offered for participation in other NCES studies, such as the ECLS-K, ECLS-K:2011, TIMSS, and the Program for International Student Assessment (PISA). However, based on previous difficulties in recruiting schools for the originally approved MGLS field test recruitment, and the general decline in school participation in NCES longitudinal studies over the years,⁶ we propose to also test offering one third of the sample schools $400 (Condition 2), and one third of schools a choice of one of seven non-monetary incentives equivalent to $400 (Condition 3). The list of the non-monetary incentive choices is provided in Table 4.

The school incentive experiment, with the same three experimental conditions, will be repeated during the MGLS:2017 Operational Field Test (OFT), which will be conducted in January through June 2017 and which will follow the same recruitment procedures as the IVFT.

The purpose of the IVFT is to test the instruments on at least 1,200 students in each of grades 6 through 8, 350 students in grade 5, and at least 200 respondents in each of three disability groups: specific learning disability, autism, and emotional disturbance. To achieve this goal, the number of participating schools in the IVFT should be at least 58 schools. A convenience school sample of about 250 schools will be selected for the IVFT from which to recruit the 58 schools. This not only assures the attainment of the requisite number of participating schools but also provides increased power to the previously proposed school incentive experiments. The larger school sample accounts for the challenge of securing school participation for the IVFT, given the brevity of the period between the start of recruitment and the start of IVFT data collection (September 2015 to January 2016). As originally proposed, schools will be randomly assigned to one of three incentive treatments: $200, $400, or $400 in materials or supplies.

The 250 schools selected for the IVFT will be recruited at the same time. Of the schools who agree to participate, a selection of 58 schools representing a diversity of demographics will be included in the IVFT. All schools who agree to participate will receive their assigned incentive regardless of their selection for participation. This will enable us to fully carry out the incentive experiment with all sampled schools regardless of their selection for participation. To provide context for the next section, schools in the sample for the IVFT are considered Tier 1 schools.

Tier 2 Schools

A study of this nature has not previously been undertaken and it is unknown whether 58 schools will be sufficient to attain the desired yield of students in each of the grades and disability groups. If it is determined that additional schools, beyond the 58, are needed to achieve the desired student yield within each of the subgroups, additional schools (Tier 2 schools) will be recruited to participate in only the student component of the study. For the purpose of the IVFT, collecting data from school staff and parents in the 58 participating sample schools should be sufficient to inform the operational field test and main study questionnaire testing. Thus, only students will be assessed in Tier 2 schools beyond the initial 58 participating schools to achieve the desired yield targets.

The opportunity for their students to participate in field-testing assessments for a national study is sometimes of considerable interest to school officials, and securing such “as needed” volunteer schools will safeguard the success of the IVFT. For these as-needed volunteer schools, depending on the school configurations and needs of the IVFT, participation may also be restricted to a subset of grades (e.g., one school may volunteer to have only their 5th-graders participate and another school may ask that only 8th-graders be included).

Tier 2 schools will be identified through a variety of means including the following activities:

School officials (and district officials, if applicable) may provide positive response to volunteer participation requests made by middle-grades research and policy community organizations and representatives, including the Association for Middle Level Education (AMLE) and the National Forum to Accelerate Middle-Grades Reform (the Forum). There will be MGLS:2017 study representation (including the NCES project officer and RTI associate project director) and visibility (an exhibit booth and study update presentation) at the AMLE annual conference in October 2015 to provide information about the study, which may provide a mechanism for schools and districts to express their interest.
Project personnel may identify volunteer schools through networking means, based on professional and personal relationships with various school- and district- officials.
District officials that agree for their sampled schools to participate in the study may offer to have additional school(s) in their district included if needed.
School officials at tier 2 schools may suggest additional schools that might be potential tier 2 volunteers.

Tier 2 schools will be considered an as-needed reserve pool of schools, and their participation in the IVFT will depend on the student yield overall and by various categories (e.g., grade level, disabilities oversample, school characteristics, and student characteristics).

It is estimated that 250 (Tier 1) schools will be recruited in the IVFT and 125 schools in the OFT in order to yield the 50 to 58 schools that will participate in each data collection. The IVFT and OFT school incentive experiment data will be combined for analysis, increasing the analytic sample size to approximately 375 sample schools. To control for field test membership, a variable indicating the field test to which the school belonged will be included along with an interaction term.

Table 4. Non-Monetary Incentive Choices for Schools in Experimental Condition 3

Incentive	Value
Registration for Association for Middle Level Education (AMLE) or Regional Annual Meeting	$400
Two-Year School Membership in AMLE	$400
Membership in Regional ML Organization plus Subscriptions to Professional Journals	$400
Professional Development Webinar	$400
School Supplies	$400
Library of Middle Level Publications	$400

School coordinators will be offered a $150 monetary incentive. They play an especially important role in the study and are critical to its success. The coordinator in each participating school will coordinate logistics with the data collection contractor; compile and supply to the data collection contractor a list of eligible students for sampling; communicate with teachers, students, and parents about the study to encourage their participation; distribute and collect parental consent forms; and assist the test administrator in ensuring that the sampled students attend the testing sessions. As described above for schools that agree to participate but are not selected for participation, the school coordinators in these schools will also receive the incentive for the work performed prior to learning that their school would not be selected (e.g., providing student list for sampling and coordinating other logistics for the data collection).

A.10 Assurance of Confidentiality

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (20 U.S. Code, Section 9543). By law, the data provided by schools, staff, parents, and students may be used only for statistical purposes and may not be disclosed or used in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573). The laws pertaining to the collection and use of personally identifiable information will be clearly communicated in correspondence with states, districts, schools, teachers, students, and parents. Letters and informational materials will be sent to parents and school administrators describing the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential. A request for a list of middle grade students with IEPs will be requested from school districts and/or schools under FERPA exception (34 CFR Part 99.31). This information will be used for sampling purposes only and will be securely destroyed once student samples are drawn.

The confidentiality plan developed for the MGLS:2017 requires that all contractor and subcontractor personnel and field workers who will have access to individual identifiers sign confidentiality agreements and notarized nondisclosure affidavits. The plan also requires that all personnel receive training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. NCES understands the legal and ethical need to protect the privacy of the MGLS:2017 respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The data files, accompanying software, and documentation will be delivered to NCES by the data collection contractor at the end of the project. Neither names nor addresses will be included in any data file.

A.11 Sensitive Questions

The recruitment effort does not involve gathering information considered to be of a sensitive nature. A list of middle-grade students with IEPs will be requested from school districts, or schools where appropriate, under FERPA exception (34 CFR Part 99.31). This information will be used for sampling purposes only and will be destroyed once student samples are drawn. All district and school personnel facilitating the conduct of the study and developing the sampling frame will be informed of the privacy and confidentiality protocols required for the study, including those having to do with the sample lists of schools and students.

A.12 Estimates of Burden

Table 5 shows the expected burden for districts, schools, and parents during the IVFT recruitment activities.

As shown in Part B, we anticipate contacting approximately 250 Tier 1 schools to reach the approximately 58 schools needed for participation, and contacting the parents of approximately 6,172 students to yield approximately 3,950 participating students. In order to draw samples of students with disabilities, we may need to obtain student records information from up to four districts. We anticipate needing to contact up to 12 districts to gain participation from four.

We estimate that it will take 10 minutes on average for school and district administrators to review the materials and either agree or decline to participate. In addition, 10 minutes has been added to the original burden estimates for all districts and schools to debrief on reasons why schools or districts chose to participate or not to participate in the IVFT. The debrief may be completed via phone or email and will ask school administrators what factors led to the decision to participate or not to participate in MGLS:2017. For example, the debrief may ask whether content, incentives, time, and/or other factors impacted the decision. The information would only be collected from tier 1 schools and will be used to inform recruitment efforts for the OFT and the main study. For those participating, we estimate an additional 4 hours for the provision of student rosters, including information about students for sampling, contact information for their parents, and their math and special education teachers (see Appendices S and T). For student’s parents, we estimate that it will take up to 10 minutes to review the recruitment materials and either consent or refuse to participate (on behalf of their student and themselves). The provision of student rosters and the parents’ consent forms will serve as sources for parents contact information, which during the data collection period can be used for nonresponse follow-up.

Table 5. Item Validation Field Test (IVFT) Recruitment Burden Estimates for Schools and Parents

Recruitment	Number of respondents and responses	Average burden time (minutes)	Total burden (hours)	Respondent average hourly wage¹	Estimate of respondent labor cost
Nonparticipating districts	8	20	3	$44.13	$132
Participating districts	4	260	17	$44.13	$750
Nonparticipating schools	192	20	64	$44.13	$2,824
Participating schools	58	260	251	$44.13	$11,077
Tier 2 participating schools	20	180	60	$44.13	$2,648
Students’ parents	6,172	10	1,029	$22.71	$23,369
Total	6,454	-	1,424		$40,800

Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/datatype: Occupation codes: Parents - All employees (00-0000) and Education Administrators (11-9032); accessed on June 18, 2015.

A.13 Total Annual Cost Burden

There are no respondent costs other than the cost associated with response time burden.

A.14 Annualized Cost to Federal Government

The estimated cost to the federal government for contractor and subcontractor work to conduct all aspects of the IVFT will be provided in the data collection submission in September 2015 (OMB# 1850-0911 v.4).

A.15 Program Changes or Adjustments

In June 2014, NCES obtained OMB clearance for district and school recruitment for the MGLS:2017 Item Validation Field Test. The recruitment was scheduled to begin in the summer of 2014 with data collection beginning in February 2015. NCES has since revised the schedule and plan for the MGLS:2017 developmental work. The IVFT will be conducted starting in January 2016, with school recruitment beginning in September 2015, as soon as OMB approval of the change request is received. As a result, NCES requested to discontinue the original approval for recruitment (OMB# 1850-0911 v.1), and this submission seeks reinstatement of the 1850-0911 OMB number with approval for recruitment for the IVFT.

A.16 Plans for Tabulation and Publication

The results from the IVFT will be presented in a report released approximately 6 months after the completion of the field test.

Table 6. Schedule for Item Validation Field Test

Activity	Start date	End date
Recruitment of schools and districts	September 2015	March 2016
Recruitment of students and parents through requesting parent consent from parents	October 2015	May 2016
IVFT Data Collection	January 2016	June 2016
Field Test Report	‒	December 2016

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all recruitment materials.

A.18 Exceptions to Certification Statement

No exceptions to the certification statement are requested or required.

1 Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G., Hagedorn, M.C., Daly, P., and Najarian, M. (2012). Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011), User’s Manual for the ECLS-K:2011 Kindergarten Data File and Electronic Codebook (NCES 2013-061). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

2 Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G., Hagedorn, M.C., Daly, P., and Najarian, M. (2001). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), User’s Manual for the ECLS-K Base Year Public-Use Data Files and Electronic Codebook (NCES 2001-029). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

3 Ingels, S.J., Pratt, D.J., Herget, D.R., Burns, L.J., Dever, J.A., Ottem, R., Rogers, J.E., Jin, Y., and Leinwand, S. (2011). High School Longitudinal Study of 2009 (HSLS:09). Base-Year Data File Documentation (NCES 2011-328). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

4 Retrieved from http://www.select-statistics.co.uk/sample-size-calculator-two-proportions.

5 Ingels, S.J., Pratt, D.J., Herget, D.R., Burns, L.J., Dever, J.A., Ottem, R., Rogers, J.E., Jin, Y., and Leinwand, S. (2011). High School Longitudinal Study of 2009 (HSLS:09). Base-Year Data File Documentation (NCES 2011-328). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

6 For example, in 1998–99, the Early Childhood Longitudinal Study had a weighted school-level response rate of 74 percent, whereas 12 years later, the complementary ECLS-K:2011 study had a weighted school-level response rate of 63 percent.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	anon
File Modified	0000-00-00
File Created	2021-01-24

Part A MGLS 2017 IV Field Test 2016 Recruitment