Carried over Part B Field Test

Part B of this submission presents information on the collection of information employing statistical methods for the Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) Item Validation Field Test (IVFT) data collection. A separate OMB clearance request for the IVFT recruitment was approved in September 2015 (OMB# 1850-0911 v. 3, 5, and 7).

B.1 Respondent Universe

MGLS:2017 Item Validation Field Test (IVFT) will include fifth through eighth grade students and will be conducted during the 2015-16 school year, with data collection scheduled to begin in January 2016. The IVFT will use a convenience sample of schools with students representing the variability in student achievement that is required for robust psychometric analysis of the MGLS:2017 reading, mathematics, and executive function assessment, and will also aim to maximize geographic diversity to the extent possible.

While not required to be nationally representative for psychometric analysis, the IVFT will include students in the typical age range found in grades 6–8 in the United States and in the typical range of abilities on the constructs being measured by the MGLS:2017 item pool. To help ensure that the mathematics assessment used during the 2017-18 base year will be appropriate for sixth grade students across their range of achievement levels, including for students with lower achievement levels in mathematics, the IVFT sample will also include a selection of fifth graders. In addition, the IVFT sample will include a subset of students from three focal disability groups (specific learning disability, autism, and emotional disturbance) who are able to take standardized tests using accommodations. Schools will be selected for recruitment from the most recent available universe files: the 2013-14 Common Core of Data (CCD) to develop the public school list and from the 2013-14 Private School Universe Survey (PSS) to develop the private school list. The less restrictive sampling requirements of the IVFT mean that schools that require extensive research applications may not be included.

Students

For the IVFT to support psychometric analyses, at minimum approximately 400 student responses should be obtained per direct assessment item and 200 per survey item. To support the number of responses per item, the sample yield needs to be at least 1,200 students from each of sixth, seventh, and eighth grades. The sample will also include schools with a grade 5 so that an additional fifth grade sample of 350 students can be selected in order to examine possible floor effects of the mathematics items. Within the yield of 3,950 students, the sample will include at least 600 students in three focal disability categories (as defined in IDEA section 300.8): 200 students categorized as having a specific learning disability, 200 students categorized as having autism, and 200 students categorized as having emotional disturbance. In the IVFT, students with an Individualized Education Program (IEP), whose assessment data are required for an evaluation of differential item functioning (DIF), will not be sampled separately but will be sought first as part of the sample drawn in general education schools. If the numbers of students with IEPs are not adequate, districts may be contacted directly to aid in the identification and sampling of students with disabilities. Disability codes will be either obtained from schools as part of the initial student roster collection or from districts (see appendices S and T for student rostering materials) when the information is not available or easily obtainable from the school, or when additional students in particular disability categories are needed to meet sample size targets. If needed, in the IVFT, special schools (where more than 80 percent of students have an IEP) may be recruited to reach the target sample sizes in the three disability categories.

The field test data collection will be conducted with as many students as each school is willing to make available, with an estimated average of 120 students per school. Based on the number and grades of students the school will allow to participate in MGLS:2017, the study staff will collect rosters and draw a sample of students within grades 5-8 and of appropriately aged students in ungraded classrooms to obtain the maximum number of students permitted. Purposively, all eligible students identified as having specific learning disability, autism, or emotional disturbance will be included in the sample.

Depending on each school’s policy, either active (explicit) or passive (implicit) parental consent materials (appendices F-G) will be distributed to the sampled students, with an estimated 80 percent rate of granted consents. Among those with granted consent, an estimated 80 percent of students are expected to be present and take the assessments. Therefore, to get to a yield of 3,950 assessed students, the parents of approximately 6,172 students will need to be contacted for consent (6,172*0.8*0.8=3,950).

Schools

To achieve the desired student yield overall and in the three focal disability groups, approximately 58 participating schools will need to participate in the study.

A convenience school sample of about 250 schools will be selected for the IVFT from which to recruit the 58 schools. The large size of the school sample relative to the target yield accounts for the challenge of securing school participation for the IVFT, given the brevity of the period between the start of recruitment and the start of IVFT data collection (September 2015 to January 2016).

We will purposively select four states and the District of Columbia Metropolitan Statistical Area (MSA), selecting states to represent each of the four Census geographic regions. In the four states, we will purposively select a single MSA. We will then select 50 schools from each state’s purposively selected MSA and select 50 schools from the DC MSA, for a total of 250 schools, with the goal of obtaining 11-12 participating schools in each of the four states and the DC MSA. Some of the benefits of using MSAs over counties include:

(1) In states where districts and counties are synonymous, one does not lose the entire area if a district refuses to participate;

(2) MSAs are better able to cover all of the urbanicity types; and, like counties,

(3) MSAs involve a reasonable travel range for field staff which leads to reduced collection costs due to the clustering of schools within defined geographic areas.

The 250 schools selected for the IVFT will be recruited at the same time. Of the schools who agree to participate, a selection of 58 schools representing a diversity of demographics will be included in the IVFT. If a selected school offers grade 5, the goal will be to assess all of the fifth graders within the school. If the participating schools do not provide sufficient numbers of fifth-grade students to meet the sample targets, we will supplement the initial school sample with schools serving fifth-grade students. We will first seek schools with fifth grades in close proximity to other schools participating in MGLS. If additional schools need to be recruited to obtain the fifth-grade targets, they will be selected for cost effectiveness of field administration.

Also, if the schools do not provide sufficient numbers of students within the three focal disability categories to meet the sample yield targets, we will supplement participating schools with a sample of special schools where more than 80 percent of the students have been identified as having an IEP. We will initially seek to identify these schools in or near districts that already contain schools that have agreed to participate. If we need to recruit additional special schools, we will choose schools that allow the most cost effective field administration, given the geographic deployment of the field assessment staff.

If it is determined that additional schools, beyond the 58 from the original sample of 250 schools (referred to as Tier 1 schools), are needed to achieve the desired student yield within each of the subgroups, additional schools (Tier 2 schools) will be recruited to participate in the student component of the study. For the purpose of the IVFT, collecting data from school staff and parents in the 58 Tier 1 participating sample schools should be sufficient to inform the operational field test and main study questionnaire testing. Thus, only students will be assessed in Tier 2 schools to achieve the desired yield targets.

The opportunity for their students to participate in field-testing assessments for a national study is sometimes of considerable interest to school officials, and securing such “as needed” volunteer schools will safeguard the success of the IVFT. For these as-needed volunteer schools, depending on the school configurations and needs of the IVFT, participation may also be restricted to a subset of grades (e.g., one school may volunteer to have only their fifth-graders participate and another school may ask that only eighth-graders be included). Thus, Tier 2 schools will be considered an as-needed reserve pool of schools, and their participation in the IVFT will depend on the student yield overall and by various categories (e.g., grade level, disabilities oversample, school characteristics, and student characteristics).

Tier 2 schools will be identified through a variety of means including the following activities:

School officials (and district officials, if applicable) may provide positive response to volunteer participation requests made by middle grades research and policy community organizations and representatives, including the Association for Middle Level Education (AMLE) and the National Forum to Accelerate Middle-Grades Reform (the Forum). There will be MGLS:2017 study representation (including the NCES project officer and RTI associate project director) and visibility (an exhibit booth and study update presentation) at the AMLE annual conference in October 2015 to provide information about the study, which may provide a mechanism for schools and districts to express their interest.
Project personnel may identify volunteer schools through networking means, based on professional and personal relationships with various school- and district- officials.
District officials that agree for their sampled schools to participate in the study may offer to have additional school(s) in their district included if needed.
School officials at Tier 2 schools may suggest additional schools that might be potential Tier 2 volunteers.

Students’ Parents, Teachers, and School Administrators

In addition to collecting information directly from students, the field test will collect information directly from students’ parents and teachers. Once the student’s participation is confirmed, the student’s parent or guardian will be contacted to complete the parent survey; the student’s mathematics teacher will be contacted to complete the math teacher survey; and if the student has a special education teacher, that teacher will be contacted to complete the special education survey. Additionally, the school administrator from each participating school will be asked to complete a survey about the school, the school environment, and the administrator’s background (the content of each survey is provided in Appendix U).

B.2 Procedures for the Collection of Information

The school recruitment approach was fully described in the previous clearance submission (OMB# 1850-0911 v. 3, 5, and 7). In this submission, the methodological descriptions focus on collecting data.

The IVFT will include student surveys and direct assessments, as well as surveys for students’ parents, math teachers, special education teachers (as applicable), and school administrators. The student surveys and direct assessments will take place in the school setting and be administered using tablet computers. In order to complete the survey and direct assessment, study staff will work with schools to identify and utilize locations for administration that minimize distractions for the student and disruption to the school routine. The parent, math teacher, special education teacher, and school administrator surveys will be internet-based, so respondents will have the option to complete the survey in a variety of school and non-school settings. In order to access the internet-based surveys, parents, teachers, and school administrators will receive an email with links and instructions for completing the survey (described in more detail below).

Planning for the Data Collection Visit

During the recruitment of the school, a school staff person will be identified as a school coordinator for the study. The study team member will work with the school coordinator to verify that the students selected for the sample are still enrolled, and to identify the student’s mathematics teacher and, if applicable, the student’s special education teacher. The study team member will also work with the school coordinator to establish the following:

The schedule for data collection (i.e., days the study will be collecting data in the school, start time and end time of the school day, bell schedule for the transition between classes, and window of time during which students will be assessed during the school day);
A location in the school setting to accommodate the data collection (i.e., determining the optimal space for the study);
The required logistical information for field staff entering the school (e.g., security and parking procedures); and
The school’s preferred technique for having students arrive at and return from the study space (e.g., this may involve field staff going to classrooms to collect students, students being notified by their teacher to report to the study space, and/or students returning to a specific class on their own when finished with the assessment and survey).

A trained test administrator (TA) will conduct the student session and complete a brief (10 to 15 minutes) school facilities checklist. Completion of this instrument does not require the involvement of any school staff. The TA will complete the checklist, either online or using a paper and pencil copy of the instrument. The facilities checklist gathers information such as: the general upkeep of the school, such as the presence of graffiti, trash, or broken windows (inside classrooms, entrance and hallways, and restrooms); noise level; security measures; and the general condition of the neighborhood/area around the school (Appendix V).

Student Survey and Assessments

Student surveys and direct assessments will be administered in 90-minute group sessions during the school day. The TA will be responsible for administering the student surveys and direct assessments. A test administrator assistant (TAA) may accompany the TA to help with equipment setup and also to conduct a second session if more than one session is scheduled concurrently. The student survey and direct assessment data collection will generally be carried out as follows:

The TA (and TAA, if applicable) will arrive at the school on the appointed day of assessments 60 minutes prior to when the first assessment session begins, following all school security protocols for entering the school, and seeking out the school coordinator who will be expecting the study team field staff per the arrangements made during the planning of the data collection visit;
The TA (and TAA, if applicable) will be escorted by school staff to the designated location for the surveys and direct assessments;
The TA (and TAA, if applicable) will set up the survey and assessment space, verifying the tablets are in working order and setting them to the appropriate start screen; and
Once students arrive in the designated assessment space, the TA will read a script to introduce the survey and direct assessments, and help the students log in to begin. The survey script will also serve as informed assent for the students (appendix W).

Parent Questionnaire

The parent questionnaire is expected to take 30 minutes to complete and will feature a multi-mode approach, with self-administered internet-based questionnaires and a telephone interview follow-up for respondents not completing the questionnaire online.

Parents will have been informed of their participation in the study and of any incentive at the time of student recruitment, and parent contact information will have been collected on the consent form. The parent data collection will generally be carried out as follows:

Parent respondents will receive a letter and an email that announces the launch of the survey and provides a link to the online instrument;
Upon completion of the survey, parents will receive a thank you letter and incentive;
For nonresponding parents, follow-ups may include reminder emails, letters, or postcards with information repeating the instructions on how to access the survey. Emails will be sent approximately every 10 days and letters will be sent approximately every 3 weeks; and,
For parents who have not responded after 3 weeks, telephone calls will be initiated to encourage participation. The study team interviewer placing the telephone call may offer to complete the survey with the parent over the phone at that moment or schedule a time to follow up with the parent and complete the survey later.

Mathematics Teacher and Special Education Teacher Surveys

The mathematics teacher and special education teacher surveys are internet-based, self-administered surveys, with a paper-and-pencil option available that consists of two parts: part 1, a teacher-level questionnaire, and part 2, a series of teacher student reports (TSRs). For mathematics teachers, part 1 is expected to take approximately 20 minutes to complete, and part 2 is expected to take approximately 10 minutes to complete for each student. For special education teachers, part 1 is expected to take approximately 10 minutes to complete, and part 2 will take approximately 25 minutes to complete for each student. The mathematics and special education teacher survey data collection will generally be carried out as follows:

For all students, their math teacher, and for students identified as having a special education teacher, their special education teacher will receive an email announcing the launch of the survey, and providing them with a link to the survey;
Upon completion of the survey, teachers will receive a thank you letter and incentive;
While at the school to conduct the student sessions, TAs will leave hand-written notes in the teachers’ mailboxes reminding them to complete their survey and thanking them if they have already participated;
For nonresponding mathematics or special education teachers, follow-ups may include reminder emails, letters, or postcards with information repeating instructions on how to access the survey. Emails will be sent approximately every 10 days and letters will be sent approximately every 3 weeks; and,
For teachers who have not completed their internet-based surveys after approximately 3 weeks, additional follow-ups may be used, including but not limited to, a telephone call encouraging teachers to complete their surveys (with the offer to conduct the survey with the teacher over the phone at that moment or scheduling a time to call back and conduct the survey later).

School Administrator Questionnaire

The school administrator questionnaire will be web-based and self-administered. It will take the administrator (generally, the principal or principal’s designee) approximately 20 minutes to complete. The school administrator data collection will generally be carried out as follows:

School administrators will receive an email announcing the launch of the survey with a link to the survey;
Upon completion of the survey, school administrators will receive a thank you email including information about how the school will receive its full incentive;
While at the school to conduct the student sessions, TAs will ask to meet with the school administrator to thank them for the school’s participation and remind them to complete the survey if they have not done so already. TAs unable to meet with the school administrator personally will leave hand-written notes in the school administrator’s mailbox as a reminder to complete their survey and thanking them if they have already participated;
For nonresponding school administrators, follow-ups may include reminder emails, letters, or postcards with information repeating instructions on how to access the survey. Emails will be sent approximately every 10 days and letters will be sent approximately every 3 weeks; and,
For school administrators who have not completed their internet-based survey after approximately 3 weeks, additional follow-ups may be used, including but not limited to, a telephone call encouraging school administrators to complete their survey (with the offer to conduct the survey with the administrator over the phone at that moment or scheduling a time to call back and conduct the survey later).

For examples of the communication to respondents inviting participation to the surveys, see Appendix W.

B.3 Maximizing Participation

A major challenge in any survey today is obtaining high response rates. More importantly, the success of the field test is contingent upon having sufficient information (enough responses per item) on the assessment and survey items to make psychometric decisions.

The data collection plan approaches the school as a community. We aim to establish rapport with the whole community—principals, teachers, parents, and students. The school community must be approached with respect and sensitivity to achieve high levels of participation. Test Administrators will be trained in all tasks, from securing school and teacher cooperation to completing child assessments and following up with parents. This approach provides continuity in contact with the school community and helps build rapport with all types of respondents. In addition to sound data collection approaches, the study will also offer monetary incentives to schools (described in Part A, Section A.9), which have proven in other NCES studies to increase school participation rates. Along with offering monetary incentives, our plan for maximizing district, school administrator, and parent engagement includes the following:

Experienced recruiters and test administrators. The recruiting and test administration teams will include staff with established records of successfully recruiting and conducting data collection in school districts and schools. To maximize district approval, senior staff will make the initial district telephone contacts. Their familiarity with the study and its future impact, as well as their experience in working with districts to overcome challenges to participation, will be crucial to obtaining district approval. Recruiters contacting schools will be equally adept at working with school administrators and providing solutions to overcome the many obstacles associated with student assessments, including conflicts related to scheduling and assessment space, minimizing interruption to instructional time, and obtaining teacher and parent buy-in. Test administrators will demonstrate flexibility in working with the school and assuming as much of the burden as possible while conducting the student sessions.
Persuasive written materials. Key to the plan for maximizing participation is developing informative materials and professional and persuasive requests for participation. The importance of the study will be reflected in the initial invitations from NCES (Appendices C-G) sent with a comprehensive set of FAQs (Appendix I) and a colorful recruitment-oriented brochure describing the study (Appendix H). Reviewing these study materials should provide districts and school administrators with a good understanding of the study’s value, the importance of the field test, and the data collection activities required as part of the study. A full understanding of these factors will be important both to obtain cooperation and ensure that schools and districts accept the data collection requests that follow.
Persuasive electronically accessible materials. In addition to written materials, we will develop a recruitment-focused website which, drawing heavily on the written materials, will present clear and concise information about the study and convey the critical importance of participating in it. Middle grades membership organizations, such as AMLE and the Forum, will provide an outreach service, asking for support of the study, offering updates to their constituencies on the progress of the study, and making available information on recent articles and other material relevant to education in the middle grades.
Buy-in and support at each level. During district recruitment, the study team will seek not just permission to contact schools and obtain student rosters but also to obtain support from the district. This may take the form of approval of a research application and a letter from the district’s superintendent encouraging schools to participate. Active support from a higher governing body or organization, such as a district or a diocese, encourages cooperation of schools. Similarly, when principals are interested in the research activity, they are more likely to encourage teacher participation and provide an effective school coordinator.
Avoiding refusals. MGLS:2017 recruiters will work to avoid direct refusals by focusing on strategies to solve problems or overcome obstacles to participation faced by district or school administrators. The recruiters will endeavor to keep the door open while providing additional information, demonstrating flexibility in the timing of administration, and identifying targeted solutions to address school concerns.

Additionally, the MGLS:2017 study team proposes to take at least the following several steps to increase response rates as part of the IVFT data collection:

The data collection plan minimizes the time that any one student, parent, or teacher will be asked to spend in completing survey instruments. For example, the student survey and direct assessment was designed to take approximately 90 minutes per student. The parent interview was designed to take an average of 30 minutes. The mathematics teacher survey was designed to take approximately 20 minutes for the teacher-level questions and approximately 10 minutes (per student) for the teacher-reported questions about students. The special education teacher survey was designed to take approximately 10 minutes for the teacher-level questions and approximately 25 minutes (per student) for the teacher-reported questions about students.
Internet-based surveys and other computer-assisted methods will be used to collect data from parents, teachers, and school administrators, while offering alternative modes for nonrespondents. The use of internet-based surveys for teachers and school administrators is quite common, but relatively few national studies have used the online surveys to collect data from parents of children in the middle grades. The field test will provide NCES with important information on the percent of parents who choose to respond using the different modes, item completion rates in each mode, and the level of effort that is required to obtain a response.
A monetary incentive will be offered to schools, parents, and teachers to encourage their participation and thank them for their time (for more information on incentives see Part A, Section A.9).
A variety of communication materials (advance letters, email invitations, a study summary, and follow-up letters) will be sent to respondents to communicate the importance of the study and of their participation, and how their participation will inform education policy for the future.
Contact will be maintained with respondents using various approaches and through multiple attempts. By staying in contact with reluctant respondents to determine their primary reasons for not responding, the data collection staff can be flexible and proactive. Direct contact with respondents by phone after unsuccessful email and hardcopy invitations can often break through resistance and help to increase cooperation and response rates. Experience with each of these modes during the IVFT will help to inform the design and data collection protocol for the main study.

B.4 Purpose of Field Test and Data Uses

The IVFT is designed to obtain the cooperation of enough schools in order to sample and collect sufficient data from students, parents, teachers, and administrators to evaluate the psychometric properties of the assessment and survey items and the predictive potential of scale variables. This evaluation will allow valid, reliable, and useful assessment and survey instruments to be developed for the main study, with the OFT serving as a second opportunity to test all assessments and surveys and all procedures one year in advance of the main study. Additionally, as described in Part A, section A.9, the IVFT will include a school and a parent incentive experiment to determine whether different levels and/or types (in the case of schools) of incentives can significantly improve participation rates.

B.5 Individuals Responsible for Study Design and Performance

The following individuals at NCES are responsible for the MGLS:2017: Carolyn Fidelman, Gail Mulligan, Chris Chapman, and Marilyn Seastrom. The following individuals at RTI are responsible for the study: Dan Pratt, Debbie Herget, Steven Ingels, and David Wilson, along with subcontractor staff: Sally Atkins-Burnett (Mathematica) and Michelle Najarian (ETS).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-24

Carried over Part B Field Test

Carried over - Part B MGLS 2017 IV Field Test 2016.docx

Middle Grades Longitudinal Study of 2017-2018 (MGLS:2017) Recruitment for 2017 Operational Field Test