Part A MGLS 2017 Field Test District & School Recruitment

This package requests clearance to recruit school districts and schools for the spring 2015 field test for the Middle Grades Longitudinal Study of 2016-2017 (MGLS:2017). MGLS:2017 will be the first study sponsored by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), to follow a nationally-representative sample of students as they enter and move through the middle grades (grades 6-8). The data collected through repeated measures of key constructs will provide a rich descriptive picture of the academic experiences and development of students during these critical years and will allow researchers to examine associations between contextual factors and student outcomes. There is a wealth of research highlighting the importance of mathematics and literacy skills for success in high school and subsequent association with later education and career opportunities, thus the study will focus on student achievement in these areas along with measures of student socioemotional wellbeing and other outcomes. The study will also include an oversample of students with different types of disabilities that will provide descriptive information on their outcomes, educational experiences, and special education services.

Baseline data for the MGLS:2017 will be collected from a nationally-representative sample of 6^th grade students in spring of 2017 with annual follow-ups in spring 2018 and spring 2019 when most of the students in the sample will be in grades 7 and 8, respectively. In preparation for the national study, the data collection instruments and procedures must be field tested. NCES has contracted with Decision Information Resources, Inc. and its partners Mathematica Policy Research and Educational Testing Service to conduct the field test in school year 2014-2015.

NCES requests permission to contact and recruit public school districts and public and private schools to participate in the field test. Districts will be contacted for the purpose of obtaining their cooperation and assistance in sampling schools for the field test. Schools that are chosen to participate will be contacted to obtain their cooperation and to gather information that will be used to facilitate the selection of students and to plan for data collection visits to the field test schools. In this package, we present the field test sample design, including the plans for selecting districts, schools, and students, and the procedures for recruiting districts and schools in 2014. This package also provides an overview of the field test, including its design and data collection procedures. A separate package will be submitted to request clearance for the field test data collection, including selecting students for the study and gathering parental consent for students to participate, securing the participation of parents, teachers, and school administrators, data collection instruments and procedures, analysis plan for field test data, and schedule for reporting of field test findings. A third package requesting permission to conduct cognitive laboratories for each field test instrument was submitted to OMB for approval on March 6, 2014 (OMB# 1850-0803 v. 97). Findings from the cognitive laboratories will inform the instruments submitted with the field test data collection package.

A. Justification

A.1 Circumstances Necessitating Collection of Information

The years between ages 10 and 14 are marked by dramatic changes in multiple domains of development. At that same time, students are experiencing many changes in their educational environments, for example, changing schools when transitioning from elementary to middle school, being taught by multiple discipline-specific teachers, and aging out of after-school child care programs. However, most research focused on this developmental stage and on the education that adolescents receive during this period of life has relied on data from small, often non-representative samples. These existing studies provide consistent evidence of declines in academic motivation, interest in school, and achievement across the early adolescent years—particularly as adolescents make the transition to middle school or junior high school (see Eccles et al., 1993; Wigfield, et al, 2006), marking this as an important educational juncture. The decline in academic motivation and disengagement from school is particularly notable for youth and adolescents who have already experienced educational or socioemotional troubles and individuals coming from lower socioeconomic and disadvantaged communities and families (see Wigfield et al., 2006). Despite growing evidence attesting to the importance of this developmental stage for understanding the perpetuation and production of educational and subsequent social inequality, there is generally a lack of diverse, nationally representative data to assist researchers and policymakers in understanding the individual, social, and contextual factors that influence youth outcomes. The federal government is uniquely positioned to undertake the needed comprehensive large-scale longitudinal study of a nationally representative sample of youth that includes measures of known critical influences on adolescents’ academic and socioemotional trajectories. NCES is authorized to conduct MGLS:2017 under U.S.C. 20, Section 9543 (Education Sciences Reform Act of 2002).

National Study Design. The MGLS:2017 aims to obtain an understanding of students’ development and learning that occurs during the middle grade years (grades 6 through 8) and that is predictive of future success, along with the individual, social, and contextual factors related to that positive development. A key goal of the study is to provide researchers and policymakers with the information to better understand the school and non-school influences associated with better mathematics and reading success, socioemotional health, and positive life development during this age period and beyond (for example, high school graduation, college and career readiness, healthy lifestyles). Examples of the key questions the MGLS:2017 is being designed to answer include:

How do students develop academically in the middle grades, and what factors are related to this development?
What is the nature and extent of mathematics learning in the middle grades?
What is the nature and extent of reading acquisition in the middle grades?
What factors (e.g., poverty, low parent education, and family structure) are related to academic success of students in the middle grades?
What instructional and school factors are associated with more positive growth in mathematics ad reading in the middle grades?
How is academic achievement in mathematics and reading related to other student outcomes, such as socioemotional well-being and executive function?

The study is also being designed to answer the above-set of questions for students with different types of disabilities. For example, does the strength of the associations between mathematics and reading achievement and students’ executive function differ by disability type? In addition, the study will provide answers to questions that are specific to students with disabilities, such as:

To what extent are students with disabilities included in general education classrooms in middle school, and to what extent do they participate in general education curricula? Do these characteristics of their education change over time, and how?
To what extent do parents participate in the IEP process and how satisfied are they with the services their children are receiving?
What are students’ IEP goals, and how is progress towards these goals measured?
For students who cannot participate in the standard MGLS:2017 assessments, how do their teachers rate their life skills (e.g., adaptive behaviors and independent functioning) and academic functioning at grades 6, 7, and 8?

MGLS:2017 serves two main purposes: (1) to provide a rich descriptive picture of the experiences and lives of middle grades students and (2) to examine the associations between student outcomes and contextual factors during this time of transition from elementary to high school. To achieve this broad set of goals, MGLSL:2017 requires a set of longitudinal and complementary instruments across several types of respondents. Collecting data over time and from multiple sources will provide information on the outcomes, experiences, and perspectives of students across grades 6, 7, and 8; their families and home lives; their teachers, classrooms, and instruction (with a focus in mathematics); and the school settings, programs, and services available to them. At each wave of data collection in the national study, students’ mathematics and reading achievement, socioemotional development, and executive function will be assessed. Students will also complete a survey that asks about their engagement in school, out-of-school experiences, peer group relationships, and identity development. Parents will be asked about their background, family resources, and involvement with students’ education and their schools. Students’ mathematics teachers will complete a two-part survey. In part 1, teachers will be asked about their background and classroom instruction. In part 2, teachers will be asked to report on the academic behaviors, mathematics performance, and classroom conduct of each study child in their classroom. For students receiving special education services, their special education teacher or provider will also complete a survey questionnaire similar in structure to the two-part mathematics teacher instrument, consisting of a teacher-level questionnaire and student-level questionnaire, but with questions specific to the special education experiences and services of the study student. School administrators will be asked to report on school programs and services, as well as school climate. The Common Core of Data (CCD) and Private School Survey (PSS) will be used as sources for information on school (e.g., school type, grade configuration) and student-body characteristics (e.g., percent minority enrollment and percent of students receiving free- or reduced-price lunch). Student information such as attendance, grade retention, and scores on state and national standardized assessments will be abstracted from school records. Lastly, field staff will complete an observation checklist on the physical condition and general upkeep of the school, the presence of different security measures, and handicap access.

Field Test Study Design. Prior to the spring 2017 national data collection, a MGLS:2017 field test will be conducted to: (1) evaluate the proposed assessments and instruments, and (2) evaluate the data collection procedures in a real world setting, identify potential challenges that could hinder the national effort, and test various procedures and strategies to inform plans for the national study. The field test will provide information about: (a) whether the selected assessment items and key non-assessment measures will allow us to answer the intended research questions, (b) whether the measures of key constructs are psychometrically sound, and (c) whether the survey instruments are appropriate for use across diverse samples of middle grade students (including students from different language groups and language proficiencies and those with different disabilities), teachers, administrators, and parents. It will guide final item selection and revisions and help meet the goal of developing valid and reliable assessments to measure middle grade students’ mathematics and reading achievement and executive function skills, as well as their socioemotional well-being and other developmental and contextual factors. Finally, the field test will serve to evaluate procedures to be used to select, recruit, and gather data from schools, students, teachers, and parents, and determine if modifications to these procedures are needed to avoid any challenges identified during the field test when the national study is undertaken. The field test will be used to identify effective procedures for use in the national study to identify, sample, and obtain reliable data on students with different disabilities.

The MGLS:2017 field test will include students in grades 5 through 8 attending public or private schools in the contiguous United States. It will also include an oversample of students with three types of disabilities (specific learning disability, emotional disturbance, and autism). We expect to conduct the field test in approximately 50 schools in five sites (a site is comprised of a geographic area with one or more counties or county equivalents in the contiguous United States or the District of Columbia and its surrounding counties.) and to collect data from an estimated 4,075 students, including 250 students in the targeted categories of students with disabilities. The field test will also survey up to 450 mathematics teachers, 150 special education teachers, 819 parents, and 50 school administrators. Field test data collection will include the same instruments and activities that are planned for the national study. Students will be administered a set of standardized assessments or item sets to measure their mathematics and reading achievement, as well as executive function, and will be asked to complete a student questionnaire¹ to gather data on their socioemotional well-being, social behavior, as well as other school and outside-of-school experiences. Their parents/guardians will be asked to complete a questionnaire that captures information about the students’ families, and their engagement in their child’s education. Mathematics and special education teachers will be asked to complete a two-part questionnaire: a teacher-level questionnaire about the teacher’s backgrounds and experience, and a student-level questionnaire about the teacher’s rating of the study student’s performance and skills. School administrators will be asked to answer questions about the characteristics of their school’s population, staffing, programs, and academic support resources. Student transcripts will be collected to gather information on student outcomes, such as grades, academic honors, and disciplinary actions. Field staff will record their observation of the school facilities and grounds. Table 1 includes a summary of the field test instruments, with expected sample sizes, modes of administration, and estimated administration times. However, this request is only for recruiting districts and schools to participate in the field test.

Table 1. MGLS:2017 Field Test Instruments, Target Sample Yield, Estimated Administration Time and Mode

Instrument	Target Sample Yield	Target Disability Yield	Administration Time	Mode
Mathematics Assessments grades 6, 7, 8 ^a	4,075	250	30 minutes; 45 minutes for students with disabilities	Group administered with computer
Reading Assessments grades 6, 7,8^a	4,075	250	30 minutes; 45 minutes for students with disabilities
Executive Function Measures	819	300	10 minutes
Student Questionnaire (including SE measure)	819	300	25 minutes
Parent Interview	819	300	30 minutes	Web, paper, telephone follow-up
Mathematics Teacher Questionnaire	450	150	23 minutes
Mathematics Teacher Student Report (TSR)	819	300	10-13 minutes per student review
Special Education Teacher Questionnaire	150	150	20 minutes
Special Education Teacher Student Report	300	300	10 minutes per student review
School Administrator Survey	50	50	33 minutes

^aThe student sample for the mathematics and reading assessments will include 350 fifth graders to guard against floor effects.

District and School Recruiting

Recruiting schools for the MGLS:2017 field test will be a two-step process. First, for public schools and some charter schools we will secure agreement from the district, and then proceed to recruit individual schools. For most private schools, recruitment will begin at the school level, although if the school operates under a diocese or other administrative unit or consortium, we will contact that administrative unit prior to contacting schools.

District Recruitment

The field test will be conducted at 5 sites with each site containing at least one regular public school district and a private school serving grades 6-8.² (If there are multiple districts in the site that are large and diverse enough to satisfy the minimum requirements of the field test, one district will be designated as the “main” district to recruit for the study and the others will be used as replacements if the main district declines to participate.

Upon receipt of OMB approval for the recruitment activities, the superintendent in each of the selected public school districts (or the main district) in the 5 sites will be sent a field test introductory packet that will include an introductory letter from ED (Appendix A), Frequently Asked Questions about the MGLS:2017 field test (Appendix C), and a MGLS:2017 Field Test Summary (Appendix D). Senior project staff leading district recruitment will conduct a follow-up call to each district to confirm receipt of the recruitment package, provide additional context for the study and data collection requirements, and to secure the district’s permission to contact schools. During the call, the recruiter will answer any questions the superintendent or other district staff have about the field test, review the list of schools in the district serving grades 6-8, and confirm key information about the schools (e.g., grades served, size of enrollment). Information collected during this call will be used to confirm schools’ eligibility for participation in this study, and collect relevant information about school leaders that might be useful for recruitment. These recruitment calls will also be used to collect information about school enrollments of students with disabilities and to identify any barriers we may face in collecting student’s IEP information, which is needed for the field test oversample of this group of students.³ NCES has obtained a FERPA exemption that will allow us access to information on students’ IEPs for sampling purposes. Recruiters will also determine if there are any additional requirements at the district level to conduct research in their schools (i.e., IRB review process) and whether the district has any restrictions on providing school staff with incentives to participate in the study.

Wherever possible, recruiters will approach the district with a goal of gaining both permission and support. They will ask superintendents to let their principals know that a study representative will be contacting them and that the district has approved this contact, as well as ask for a letter of support from the district to be sent with the recruitment packet to schools. This type of district level support, where sincere and timely, greatly increases school support.

Recruiters will make multiple attempts by telephone and email to reach the superintendent over a two-week period. If we are unable to reach the superintendent after multiple attempts or if we make contact with the superintendent and the district refuses to participate and the site has more than a single school district, we will contact a second district and begin the district recruitment process again. If the second district refuses, we will attempt to convert the district whose refusal was softer or where our contact remained amicable. Where this fails, we will break-off contact with the superintendents and begin again the recruitment process using one of the two replacement sites.

Once district cooperation is obtained and the school sample is selected, we will complete any district-level paperwork, including completing a district IRB application, and begin the school recruitment process once we have final approval from the district to do so.

The approach to recruiting private schools and some charter schools may or may not follow the same approach outlined for district recruitment. If the private schools selected for the field test operate under a governing body such as a diocese, private school administrator or a consortium of private schools, we will use the same approach that is used for public schools. Similarly, if a charter school operates under the district, we will first contract the district for approval prior to contacting the school. However, many private schools and some charter schools will not have such a higher level governing body. In these cases, recruitment into the study will begin with school recruitment as outlined below.

School Recruitment

Once district approval (or the approval of another governing body in the case of private schools) has been acquired and the field test sample of schools chosen, a field test information packet will be sent to administrators of the field test schools. The packet will include a letter of introduction (Appendix B), a copy of the Field Test Summary (Appendix D), Frequently Asked Questions (Appendix C), and a district (or another governing body) letter of approval and support (if the district or governing body is willing to prepare one). The introductory letter will inform the school administrator that the school is being invited to participate in a field test for an important and unprecedented national study that will provide information about the middle grade years and students’ progress during this time. It will also provide an overview of what participation entails, and that it includes the participation of students in grades 6-8 (and in some schools, grade 5 students), their mathematics teachers and parents. The letter will also inform the administrator of the study’s interest in assessing and collecting data from students with disabilities, their mathematics teachers (when applicable) and special education teacher or provider, and parents. The letter will mention that a DIR staff member will contact them to discuss their school’s participation in the field test and to answer any questions they may have. Within one week, a school recruiter will follow up with a call to request the school’s cooperation and to answer any questions about the field test and what participation in the field test entails. Drawing on our experience, we plan to conduct our initial discussions with the school’s administrator (e.g., principal, headmaster) who may designate another staff member as the school coordinator, who will serve as the study’s primary contact for all future communication. If the school administrator names a school coordinator, the recruiter will contact that person to introduce the study, and discuss the role of the school coordinator as facilitator. Moving forward, the school coordinator will work with the study team to schedule data collection and field test activities at that school, including student recruitment, onsite assessments, and transcript abstraction. During our initial contact with the school, we will inform the administrator that a $500 incentive has been reserved for the school. Depending on school policy, schools may want to use part of this money to compensate the school coordinator for the key role this person will play during the field test.

During the initial school recruitment call, we will ask how many teachers provide mathematics instruction in the selected grades and confirm what type of parental consent procedures or requirements will need to be followed in order to collect data on students at the school, including for collecting information on students’ activities, grade-level enrollment, and student-teacher rosters. Information will also be collected on any considerations that may impact data collection (e.g., planned construction periods, school reconfiguration, or planned changes in leadership); and about sources for missing IEP status to supplement information collected at the district level, as we identify students with disabilities and their special education teachers. We will also gather information about hours of operation, including early dismissal days, school closures/vacations, and dates for standardized testing. This information will be important to know when scheduling the student assessments.

By design, the initial sample will include more schools in each site than we need to recruit. Schools will be randomly assigned to the initial sample (to be contacted first) and to the replacement sample (in a designated selection order). Replacement schools will be used if one of the initially sampled schools is determined to be ineligible (e.g., focal study grades no longer offered at the school) or refuses to participate. The plan for replacement will vary with the number of schools in each site; however, the plan will allow for replacement of initial sample schools with schools sharing similar characteristics (e.g., a private school would be replaced by another private school and a rural school would be replaced with another rural school).

Recruiter Training

Initial district contacts will be made by senior staff that have experience recruiting districts for other education studies. This staff can easily present the study background, purpose, and data requirements to district personnel and district research review panels. Recruiters, who will follow-up with schools and school coordinators, will also be well prepared and experienced in recruiting schools and educators for large-scale studies. They will be selected from a group of experienced research staff with past success recruiting schools for research studies, including studies sponsored by the Institute for Education Sciences (IES). They will participate in an all-day training conducted by the study design team structured to ensure that staff have a good understanding of the MGLS:2017 and the field test, what school participation entails, and that they are able to respond to questions raised during the recruitment process. The training will use a mix of lecture, group discussion, and exercises (e.g., responses to questions that school personnel may ask about the study and what participation involves). Among the topics to be covered are the purposes and design of the MGLS:2017 and the MGLS:2017 field test; the field test samples of schools, students, parents, and mathematics and special education teachers and how they are chosen; field test data collection instruments and procedures; overview of field staff’s visit to a sampled schools; the burden participation places on schools and the steps that the study team will take to reduce this burden; and common concerns raised by schools and how these will be addressed.

Monitoring

To monitor the recruitment process, team members will update a spreadsheet on the MGLS:2017 SharePoint site. Information about contact attempts (including details such as day, time, number called, and whether a message was left) and recruitment notes will help to track the status of each district and school. Disposition codes (no contact made, in process, agreed, refused) and detailed notes for all contacts will be used to monitor progress and determine whether a replacement district or school is needed, or if one of the five initial sites needs to be replaced with another one. Recruiters will be trained to provide enough detail in the notes section for the recruitment leader to assess the likely outcome while the district and school is still in process. Enough detail will be provided to enable another recruiter or the recruitment leader to follow-up. This information will be uploaded to Mathematica’s Survey Management System (SMS). The SMS will be used to track all field test recruitment and data collection activities.

DIR’s recruitment leader will meet with the recruiters once a week by phone to review the status of each district and school and identify any questions or problems that have come up in the recruitment process. Group meetings will allow recruiters to share challenges and successful strategies.

A.2 Purposes and Uses of Data

The purpose of the field test is to evaluate a battery of student assessments (i.e., mathematics and reading achievement and executive function skills) and non-assessment survey instruments (e.g., student, mathematics and special education teacher, and parent questionnaire) for use in the MGLS:2017 national study. The field test will also be used to test procedures and operations that will be used in the national study. Data gathered during the field test recruitment process will be used to validate district and school characteristics used for sample selection, including school-level information on the prevalence of students in specific disability categories (e.g., emotional disturbance and autism). The field test will provide important information about the viability of sampling these students at the district level. It will also inform recruitment strategies for the national study. The field test study team will revise recruitment materials and procedures for the national study based on the field test experience. The results of the recruitment effort will be provided in the field test report along with challenges and lessons learned.

During district recruitment, districts will be asked to review key district and school-level characteristics we have gathered for accuracy and completeness. While other sources will be used to develop current and complete data (including universe files and school and district websites), districts will have an opportunity to provide new information and correct outdated information that we collected from existent sources. For example, the recruitment team will check for recent school mergers, changes in school leadership, temporary but extended school closings for construction, and changes in grade configuration or size of enrollment. Once districts are recruited into the study, they will be asked to provide a list of students with IEPs on record, which will be used to oversample students with disabilities for the field test. If districts are willing to provide contact information (e.g., email addresses and direct telephone numbers) for principals in sample schools, we will record this information and use it during school-level recruitment.

Schools will be asked to provide guidance on the best way to gather accurate class roster information (for mathematics classes) and to identify special education teachers or providers for sampled students with disabilities. This information will be used to develop the student sample and procedures for linking students and their mathematics and special education teachers or providers.

A.3 Use of Technology to Reduce Burden

Where feasible, available technology will be used to reduce burden and improve efficiency and accuracy. For example, if districts can provide information linking students to their mathematics teachers or students with disabilities to their special education teachers electronically, we will use this information rather than asking for this information at the school level. The burden of district and school recruitment is minimal and most information is gathered over the telephone. Districts will primarily be asked to provide confirmation of data gathered from other sources, including school universe files and district and school websites. Our request to provide student lists will accommodate whatever form districts find to be the least burdensome. The study will utilize the information in whatever format it is provided.

A.4 Efforts to Avoid Duplication of Effort

The MGLS:2017 will not be duplicative of other studies. While NCES longitudinal studies have contributed to our understanding of the factors that influence student success and failure in school, the middle grades (grades 6–8) are noticeably absent from the studies conducted to date. A majority of national longitudinal studies have focused on high school students and on the transition from secondary to postsecondary education (for example, High School and Beyond [HS&B], Education Longitudinal Study of 2002 [ELS:2002]). The Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K) and the National Education Longitudinal Study of 1988 (NELS:88) collected data on samples students attending grade 8, but neither included any data collection on grades 6 and 7. The ECLS-K:2011 will not follow students beyond grade 5 and the High School Longitudinal Study of 2009 (HSLS:09) began with a national sample of students in grade 9. Thus, we have little information at the national level about the learning that occurs during grades 6-8 and about the rates of learning for different groups of students who may experience diverse school environments and opportunities.

The MGLS:2017 is unique in that it will assess student’s mathematics and reading achievement, as well as other student outcomes (for example, executive functions and dimensions of socioemotional development) for the same group of students over a three-year period. In addition to the ECLS-K and NELS:88, other national studies have assessed some of these outcomes for students in grade 8, including the National Assessment of Education Progress (NAEP) and Trends in International Mathematics and Science Study (TIMSS). These studies however are cross-sectional and do not include repeated measures of achievement, and do not assess multiple subjects and areas of development for the same sample of students. Therefore, they cannot answer questions about students’ growth in mathematics and reading over the middle grade years, differences in the rates of growth for different populations (for example, differences by gender, race-ethnicity, and for students attending public and private schools), and the school and non-school factors that may facilitate or hinder this growth. Nor can they explore questions about the relationships between student achievement and other school outcomes and executive functions (for example, working memory, attention, and inhibitory control), that work to regulate and orchestrate cognition, emotion, and behavior to enable a student to learn in the classroom.

Other adolescent development studies have been conducted, but these often do not include a grade 6 sample. For example, the youngest children in the National Longitudinal Study of Adolescent Health (Add Health) and the Maryland Adolescent Development in Context Study (MADICS) were in grade 7 at baseline. Many of these studies collected data on local samples, had a primary focus on family and child processes, and were started in the 1990s (for example, MADICS and the Michigan Study of Adolescent and Adult Life Transitions [MSALT]). As such, they do not provide a contemporary picture of grade 6-8 students in the U.S.

Within the MGLS:2017, to the extent possible, we will use existing data to develop the school and student samples rather than duplicate data collection and reporting efforts. However, while many school characteristics will be gathered from existing sources such as the CCD and PSS, districts have a unique ability to provide updates based upon more current information reflecting recent school and student enrollment information not available elsewhere. The most recent CCD and PSS data that will be available at the time the field test sample is drawn (summer 2014) are for school years 2011-2012 and 2009-2010, respectively. Districts can provide information on school-level enrollments of students with disabilities that is either not available or would require significant resources to collect. There is no universal grade 6-8 student sampling frame. We will develop the frame from information provided by districts and schools.

A.5 Methods of Minimizing Burden on Small Entities

During district and school recruitment, we will minimize burden by training recruitment staff to make their contacts as straightforward and concise as possible. The recruitment letters and materials (e.g., study description and FAQs) are designed to be clear, brief, and informative. We will include all relevant staff at any district- and school-level meetings so that the district superintendents and principals will not be required to convey the information individually to their staff members. At the district level (or at a private school organization or administrative office), we will attempt to arrange a conference call that includes representatives of the superintendent’s or director’s office and research approval office; and officials who can discuss the availability of student records. For the school-level recruitment, we will offer to organize a conference call or webinar for administrators and teachers who might be included in the study. To avoid imposing an additional time commitment and travel requirements on students participating in the field test student assessment, we will attempt to conduct all of the field test data collections at the school during the regular school day. The approaches that will be used to minimize burden for parents, teachers, students, and school administrators will be described in the second field test OMB package – Field Test Data Collection Package.

A.6 Consequences of Not Collecting Data

A field test to ensure that we have reliable instruments and effective procedures for the national study will greatly improve the quality of and enhance efficiencies in the MGLS:2017. The recruitment effort is critical, as it will offer the best opportunity to get first-hand, up-to-date information on schools to include in our field test, such as changes in school grade levels or closures and identifying students with IEPs and schools with different concentrations of these students. Collecting this information is necessary to determine school eligibility and finalize the schools that will be included in the field test sample.

A.7 Special Circumstances

There are no special circumstances involved with the recruitment and data collection.

A.8 Consultations outside the Agency

In addition to the DIR, Mathematica, and ETS study team members, several consultants have provided expert advice during the instrument development process. William Schmidt (Michigan State University) has assisted with the development of the student mathematics assessment and recommended measures and survey items for the teacher-level and student-level Mathematics Teacher Questionnaire. Jacquelynne Eccles (University of Michigan) has assisted with the selection of the executive function assessments, measures of socioemotional development, and identified content and recommended survey items for the student and parent questionnaires. Donald Rock (independent consultant, previously at ETS) has assisted with the development of the student math assessment and provided expert advice on structure of the mathematics assessment and the design of the field test sample. Kathy Terry (Houston Independent School District) has assisted with the parent and school administrator questionnaires. She is also assisting with the approaches used to secure cooperation from districts and schools for the field test. Martin Frankel (Baruch College) has assisted with the field test sample design.

The MGLS:2017 design team also convened a series of Content Review Panels (CRPs) to gather input on key constructs to measure in the MGLS:2017 and identify possible item sources. The MGLS:2017 Mathematics Assessment CRP first met in Washington DC on June 18-19, 2013, to provide input during the assessment development process and ensure that the math assessment resulted in reliable and valid data on students’ mathematics knowledge and skills. Specifically, the Mathematics CRP helped to guide decisions about which domains of mathematics are most critical to sample for the assessment. The panel also provided input on the amount of assessment time that should be devoted to different domains of mathematics and offered advice for the best ways to assess these domains given the time constraints of the MGLS:2017 assessment. Following the meeting, members of the Mathematics CRP were asked to review a revised version of the assessment framework and test blueprint and provide a short written reaction to the domains and their definitions, the learning progressions, the allocation of items within domains, and depth of knowledge within categories in the framework and blueprint. The panel members participated in a 3-hour webinar on September 13, 2013, to discuss these topics.

The MGLS:2017 Executive Function CRP met via webinar on July 18, 2013, to provide input during the assessment development process and help guide decisions about which constructs are most critical to examine. The panel provided input on potential measures and scales for assessing middle grades students’ executive functioning (for example, inhibitory control, cognitive flexibility, and working memory).

The MGLS:2017 Socioemotional-School-Family CRP met in person for a 2-day meeting in Washington DC on July 25-26, 2013, to provide input during the instrument development process and ensure that the assessment and questionnaires will result in reliable and valid data on students’ socioemotional well-being, school experiences, and home life. The panel provided input on potential items and scales as well as the best reporter for various aspects of students’ socioemotional well-being (e.g., perceived competence, internalizing behavior, conscientiousness, and engagement). The panel also provided input on the student and parent questionnaires that the MGLS:2017 will use to collect data on students’ family, school, and out-of-school experiences.

Finally, the MGLS:2017 School Administrator CRP met via webinar on August 16, 2013, to provide input on constructs to measure school-level characteristics. The panel provided input on potential items and scales as well as the best reporter for various constructs.

Following the CRP meetings, a revised construct matrix was sent out to panelists for comment. They were asked to provide written feedback on constructs they believed are high priority and should be included in the field test questionnaires, in which waves (i.e. in grade 6, 7, and/or 8) the construct should be measured, and as reported by which type of respondent. The instruments being evaluated in the cognitive laboratory interviews reflect recommendations made by the various CRPs.

On October 31, 2013, NCES convened a panel of experts to discuss a range of issues associated with sampling, assessing, and collecting data about students with different disabilities. Another expert panel will be convened in spring 2014 to discuss these issues further with more attention on the content of the Special Education Teacher Questionnaire and student assessment accommodations. Additionally, a panel of reading expert will be convened in spring 2014 to review the plans for assessing students’ reading achievement. Input from this panel will be used to finalize the reading assessment field test instrument. The recommendations of these groups will be reflected in the second OMB package – Field Test Data Collection Package.

See Table 2 for the list of the members on each CRP that has been formed and met to date. The membership in the follow-up Disabilities CRP and the Reading CRP has not yet been finalized.

Table 2. Members of the MGLS:2017 Content Review Panels

Name	Affiliation	Expertise
Mathematics Assessment Content Review Panel
Tom Loveless	Brookings Institute	Policy, Math curriculum
Linda Wilson	Formerly with Project 2061	Math education, math assessment, and middle school assessment, author for Assessment Standards for School Mathematics (NCTM) and NAEP math framework, teacher
Kathleen Heid	University of Florida	Math education, use of technology, teacher knowledge, member of the NAEP Grade 8 mathematics standing committee
Edward Nolan	Montgomery County Schools	Math curriculum and standards, large-scale assessment of middle grade students
Lisa Keller	UMass Amherst	Psychometrics, former math teacher
Paul Sally	University of Chicago	Math education, mathematics reasoning, mathematically talented adolescents
Margie Hill	University of Kansas	Co-author of Kansas math standards, former NAEP Math Standing Committee member, former district math supervisor
Executive Function Content Review Panel
Lisa Jacobson	Johns Hopkins University; Kennedy Krieger Institute	Executive functioning; attention; neurodevelopmental disorders; parent and teacher scaffolding development of executive functioning skills
Dan Romer	University of Pennsylvania	Adolescent risk taking
James Byrnes	Temple University	Self-regulation; decision making; cognitive processes in mathematics learning
Socioemotional-School-Family Content Review Panel
James Byrnes	Temple University	Self-regulation; decision making; cognitive processes in mathematics learning
Russell Rumberger	University of California, Santa Barbara	School dropouts and ethnic and language minority student achievement
Tama Leventhal	Tufts University	Family context; adolescence; social policy; communities and neighborhood indicators
Susan Dauber	Bluestocking Research	School organization; educational transitions; urban education; parent involvement and family processes
Scott Gest	Pennsylvania State University	Social networking, social skills, and longitudinal assessment of at-risk populations.
Kathryn Wentzel	University of Maryland	Social and academic motivation; self-regulation; school adjustment; peer relationships; teacher-student relationships; family-school linkages
Richard Lerner	Tufts University	Adolescent development and relationships with peers, families, schools, and communities
School Administrator Content Review Panel
Susan Dauber	Bluestocking Research	School organization; educational transitions; urban education; parent involvement and family processes
George Farkas	University of California, Irvine	Schooling equity and human resources
Jeremy Finn	State University of New York at Buffalo	School organization, school dropouts
Edward Nolan	Montgomery County Schools	Large urban school system administrator
Tom Loveless	Brookings Institute	Policy, Math curriculum
Disabilities Considerations Content Review Panel
Lynn Fuchs	Vanderbilt University	Student assessment, mathematics curriculum, psychometrics models, learning disabilities
Margo Mastropieri	George Mason University	Learning disabilities, retention strategies, cognitive strategies, special education, research methods
Cara Laitusis	ETS Research and Development	Curriculum-based assessment and the diagnosis and treatment of students with learning disabilities.
Leslie Scott	University of Michigan	Quality/cost factors related to design of multimode school studies; web-based data collection methodologies; use of multimedia approaches

A.9 Payment or Gift to Respondents

High levels of school participation are critical to the success of the MGLS:2017 field test. School, mathematics teacher, special education teacher, parent, and student data collection activities are contingent on school cooperation. NCES recognizes that the burden level of the study is one of the factors that school administrators will consider when agreeing to participate. To offset the perceived burden, NCES intends to continue its use of strategies that have worked successfully in the past for other major NCES studies (e.g., ECLS-K, ECLS-K:2011, HS&B, NELS:88, and ELS:2002). Offering schools a financial incentive helps to offset the burden schools are asked to take on as a result of their participation. The study asks a lot of schools, including requesting a student roster with basic demographic information (e.g., date of birth, gender, and race/ethnicity); information on students’ IEP status; permission for field staff to be in the school for up to a week; administering an extensive battery of assessments to 84 students in grades 6-8; making available space for administering student assessments; allowing students to leave their normal classes for the duration of assessments; and providing information about the student’s teachers and parents. Also, on average, approximately 10 students in each school will have one of the disabilities that are of special interest to the study and many will require accommodations and different assessment settings, such as individual administration and smaller group sessions. Working with the data collection contractor to assess these students will place further burden on the participating schools.

Given the many demands and outside pressures that schools already face, it is essential that they see that MGLS:2017 staff understand the additional burden being placed on the school staff when requesting their participation. We propose to offer schools a $200 incentive for their participation. This amount is consistent with the amount of the school incentive offered by other NCES studies such as the ECLS-K, ECLS-K:2011, TIMSS, and the Program of International Assessment (PISA).

MGLS:2017 will also ask participating schools for assistance in administering the student assessments by allowing access to school computer resources and equipment, and by helping to establish internet connections to the web-based instruments. Therefore, like the HSLS:09, MGLS:2017 will offer a $50 incentive to the school’s information technology (IT) coordinator.

School coordinators play an especially important role in the study and are critical to its success. The coordinator in each participating school will coordinate logistics with the data collection contractor; compile and supply a list of eligible students for sampling to the data collection contractor; communicate with teachers, students, and parents about the study to encourage participation; distribute and collect parent consent forms; and assist the test administrator in ensuring that the sampled students attend the testing sessions. For this reason, coordinators will be offered $150 as a baseline incentive plus an additional $50 incentive if the student participation rate for the school meets or exceeds 85%. In comparison, the PISA 2015 field test offered coordinators a $200 incentive, while HSLS:09 offered $100-$150 incentive conditional on the percentage of participating students. The MGLS:2017 will sample and assess 84 students per school compared to the 42 students in PISA 2015 field test and 36 students per school in HSLS:09. The MGLS:2017 will also involve the additional effort associated with assessing an oversample of students with disabilities. The supplemental conditional $50 incentive is meant to encourage school coordinators to make the extra effort needed to overcome the fact that obtaining parent consent for middle grade students in general and for students with disabilities in particular is often challenging. The actions of the school coordinator can be central to minimizing these challenges.

All incentives will be paid by check. An honorarium check will be mailed to each school, school IT coordinator, and school coordinator at the end of spring data collection along with a note thanking each for their assistance.

A.10 Confidentiality of the Data

DIR and its research partners, Mathematica Policy Research and Educational Testing Service, will conduct all data collection, analysis and reporting activities in accordance with all relevant regulations and requirements, consistent with the Education Sciences Reform Act of 2002 (Title I, Part E, Section 183), the Family Educational and Rights and Privacy Act (FERPA) (20 U. S. C. 1232g;34 CFR Part 99), and the Protection of Pupil Rights Amendment (PPRA) (20 U.S. C. 1232h; 34 CFR Part 98).

The research team will protect the confidentiality of all data collected for the study and will use it for research purposes only. Only DIR and Mathematica Policy Research will be required to handle personally-identifying information (PII) and their project directors will ensure that all PII about respondents will be protected. All data will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. All members of the study team having access to the data will be trained and certified on the importance of confidentiality and data security. When reporting the results, data will be presented only in aggregate form, such that individuals and institutions will not be identified. Included in all requests for data will be the following statement:

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S. Code, § 9543. By law, the data provided by your schools, staff, and students may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, § 9573).

The following safeguards will be used to carry out confidentiality assurances for the study:

All employees at DIR and their partners sign confidentiality agreements that emphasize the importance of confidentiality and specify employees’ obligations to maintain it (see Appendix F).
Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by sample identification numbers.
Access to the file linking sample identification numbers with the respondents’ ID and contact information is limited to a small number of individuals who have a need to know this information
Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.
Access to computer data files is protected by secure usernames and passwords, which are only available to approved users. Access to identifying information for sample members is limited to those who have direct responsibility for providing and maintaining sample contact information. At the conclusion of the study, these data are destroyed.
Sensitive data are encrypted and stored on removable storage devices that are kept physically secure when not in use.
The plan for maintaining confidentiality includes staff training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems.

Responses will only be made available in tabular form. Under no condition will information be made available to school personnel. District and school staff responsible for assisting DIR in the data collection will be fully informed of the study’s confidentiality policies and procedures for ensuring protection and confidentiality of individual sample members and responses. The study will also adhere to state and local laws and school board policies as appropriate.

A.11 Additional Justification for Sensitive Questions

The recruitment effort is not gathering information considered to be of a sensitive nature. A request for a list of middle grade students with IEPs will be requested from school districts under FERPA exemption (February 7, 2014). This information will be considered confidential and used for sampling purposes only. All district and school personnel facilitating the conducting of the study and developing the sampling frame will be informed of the privacy and confidentiality protocols required for the study. This will include the sample lists of schools and students.

A.12 Estimates of Hour Burden

Table 3 shows the expected burden during the field test recruitment activities. We anticipate contacting up to 15 districts/diocese and 100 school administrators to recruit the final 50 schools. The expected burden for school districts/diocese is 30 minutes for those that decline and 5 hours for districts/diocese that agree to participate for a total of 35 hours of burden for school districts/diocese. The estimates for the recruitment phase of the field test for school administrators includes an estimated 30 minutes for up to 100 schools that decline or are determined to be ineligible, and 1 hour for the 50 schools that agree to participate. This is a total of 100 hours of burden for schools. The number of district and school administrators who will need to be contacted about their participation and their recruitment time is based on estimates from similar school based sample studies (e.g., ECLS-K:2011). The total number of burden hours for the field test district and school recruitment is estimated at 140 hours.

Table 3. Respondent Burden Estimates for MGLS:2017 Field Test Recruitment Activities

Recruitment	Number of Respondents	Number of Responses*	Average burden time (minutes)	Mode	Total Burden (Hrs)
Nonparticipating districts (diocese)	10	10	30	Mail, e-mail, telephone	5
Participating districts** (diocese)	7	7	300		35
Nonparticipating schools	100	100	30		50
Participating schools	50	50	60		50
Study Total***	167	167	N/A		140

*Responses include both refusals and acceptances to participate in the field test.

**For the purposes of this burden table, we have assumed one site will include 2 districts and that one site will include a district and a diocese for a total of 7 participating districts/diocese across 5 sites. We have also assumed that the 7 participating districts or dioceses will require a district level review of the proposed data collection activities by their IRB, research department, or school board. It is assumed that discussions around these requirements, review of the research application submission, and preparation of the approval notification, will total approximately 4 hours per district with an additional hour spent during the initial contact determining interest and reviewing eligibility upon acceptance.

***N/A – Not Applicable.

A.13 Estimate of Total Annual Cost Burden to Respondents or Recordkeepers

There are no additional respondent costs associated with this data collection other than the time estimated as burden.

A.14 Annualized Cost to the Federal Government

The estimated cost to the government to recruit districts and schools for the field test is $253,094.

A.15 Tabulation, Publication Plans, and Time Schedules

a. Tabulation Plans

There are no plans to tabulate information obtained from district or school personnel during school recruitment. Information about school changes or students with IEPs will be used internally for sampling purposes and to inform field test plans. The usefulness of the information requested, its accuracy, and reporting source will be documented and findings shared in the field test report.

b. Publication Plans

The field test is expected to be completed in approximately two years. School recruitment will begin in fall 2014 with data collection planned for spring 2015. Final instruments and field test reports will be provided in 2016 for the 2017 national study. When field test analyses are complete, the study team will include lessons learned during district and school recruitment in the field test report. Findings will include discussion of the information districts were able to update regarding school changes and the percentage of student enrollment with IEPs. We will report whether the information was provided by districts or schools, the timeliness with which it was obtained, and school or district burden associated with the reporting of this information. This information will be shared with and used by research staff developing procedures for the national study.

c. Schedule

Table 5 shows the schedule for the field test.

Table 4. Field Test Study Timeline of Activities

Activity	Time Period
Instrument item pool development	Spring 2013 - Summer 2014
Conduct cognitive testing for instruments*	Spring 2014 - Summer 2014
Identify sites and select school sample	Summer - Fall 2014
Recruit districts (and diocese) and schools	Fall 2014
Collect student rosters & Sample students (and their parents and teachers)	Winter 2014 – Spring 2015
Collect parent consent for student testing	Winter 2014 – Spring 2015
Data Collection	Spring 2015
Draft Field Test Report	Spring 2016
Final report and instruments for the 2017 national survey	Summer 2016

*Request for Clearance for MGLS:2017 Cognitive Laboratory Activities was submitted in March, 2014.

A.16 Reasons for Program Changes or Adjustments

This is a new project.

A.17 Exception to the Certification Statement

No exceptions to the certification statement are requested or required.

REFERENCES

Eccles, J.S, Midgley, C., Wigfield, A., Buchanan, C.M., Rueman, D., Flanagan, C., & Iver, D. (1993). Development during adolescence: The impact of stage-environment fit on young adolescents’ experiences in schools and in families. American Psychologist, 48 (2), 90–101.

Gibbons, R.D. & Hedeker, D. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423-436. Holzinger, K. J., & Swineford, F. (1937). The bi-factor method. Psychometrika, 2, 41–54.

Wigfield, A., Eccles, J. S., Schiefele, U., Roeser, R., & Davis-Kean, P. (2006). Motivation. In N. Eisenberg (Ed.), Handbook of Child Psychology (Vol. 3, 6th ed., pp. 933–1002). New York, NY: Wiley.

Wigfield, A., & Eccles, J.S. (1989). Test anxiety in elementary and secondary school students. Educational Psychologist, 24, 159–183.

1 Not every student will complete the full student questionnaire because of limits on the time that schools will allow us to work with students, and given that the field test is evaluating a larger set of measures and items than will be included in the national study. Students will be asked to complete different mathematics item sets, different measures of executive function and socioemotional development, and different sets of items from the student questionnaire.

2 Sites must also have at least 10 schools serving grades 6-8.

3 If we are able to get this information from sources other than districts, and can move forward with selecting the field test sample, we will share information about the schools selected in the districts during these initials calls.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	MGLS - Part A OMB Submission (02.11.14
Author	Mathematica Staff
File Modified	0000-00-00
File Created	2021-01-27