Part B MGLS 2017 IV Field Test 2016 Recruitment

Part B of this submission presents information on the collection of information employing statistical methods for the recruitment for the Middle Grades Longitudinal Study of 2017-2018 (MGLS:2017) Item Validation Field Test (IVFT). A separate OMB clearance request for the field test data collection will be submitted in September 2015. This document contains revisions described in the change memo to the document originally approved in July 2015 (OMB# 1850-0911 v.3)

B.1 Respondent Universe

The IVFT will include 5^th through 8^th grade students and will be conducted during the 2015-2016 school year, with data collection scheduled to begin in January 2016. The IVFT will use a convenience sample of schools representing the variability in student achievement that is required for robust psychometric analysis of the MGLS reading, mathematics, and executive function assessment, and will also aim to maximize geographic diversity to the extent possible.

While not required to be nationally representative for psychometric analysis, the IVFT will include students in the typical age range found in grades 6–8 in the United States and in the typical range of abilities on the constructs being measured by the MGLS item pool. To help ensure that the assessments that will be used during the 2017-18 MGLS base year will be appropriate for 6^th grade students with lower achievement level in mathematics, the IVFT sample will also include a selection of 5^th graders. In addition, the sample will include a subset of students from the three focal disability groups (specific learning disability, autism, and emotional disturbance) who are able to take standardized tests using accommodations. Schools will be selected for recruitment from the most recent available universe files: the 2013-14 Common Core of Data (CCD) to develop the public school list and from the 2011-12 Private School Universe Survey (PSS; the 2013-14 PSS will be used if it becomes available) to develop the private school list. The less restrictive sampling requirements of the IVFT mean that schools that require extensive research applications may not be included.

Students

For IVFT to support psychometric analyses, at minimum approximately 400 student responses should be obtained per direct assessment item and 200 per survey item. To support the number of responses per item, the sample yield needs to be at least 1,200 students from each of 6th, 7th, and 8th grades. The sample will also include schools with a grade 5 so that an additional 5th grade sample of 350 students can be selected in order to examine possible floor effects of the mathematics items. Within the yield of 3,950 students, the sample will include at least 600 students in the three focal disability categories (as defined in IDEA section 300.8): 200 students with a specific learning disability, 200 students with autism, and 200 students with an emotional disturbance. In the IVFT students with an Individualized Education Program (IEP) required for Differential Item Function study will be sought first as part of the sample drawn in general education schools. If the numbers of students with IEPs are not adequate, districts may be contacted directly to aid in the identification and sampling of students with disabilities. Disability codes will be either obtained from schools as part of the initial student roster collection or from districts (see appendix S-T) when the information is not available or easily obtainable from the school, or when additional students in particular disability categories are needed to complete the sample. If needed, in the IVFT, special schools (where more than 80% of students have an IEP) may be recruited to reach the target sample sizes in the three disability categories.

The field test data collection will be conducted with as many students as each school is willing to make available, with an estimated average of 120 students per school. Based on the number and grades of students the school will allow to participate in MGLS, the study staff will collect rosters and draw a sample of students within grades 5-8 and of appropriately aged students in ungraded classrooms to obtain the maximum number of students permitted. Purposively, all eligible students identified as having specific learning disability, autism, or emotional disturbance will be included in the sample.

Depending on each school’s policy, either active (explicit) or passive (implicit) parental consent materials (appendices F-G) will be distributed to the sampled students, with an estimated 80 percent rate of granted consents. Among those with granted consent, an estimated 80 percent of students are expected to be present and take the assessments. Therefore, to get to a yield of 3,950 assessed students, the parents of approximately 6,172 students will need to be contacted for consent (6,172*0.8*0.8=3,950).

Schools

To achieve the desired student yield overall and in the three focal disability groups, we will need to recruit approximately 58 schools.

A convenience school sample of about 250 schools will be selected for the IVFT from which to recruit the 58 schools. The size of the school sample accounts for the challenge of securing school participation for the IVFT, given the brevity of the period between the start of recruitment and the start of IVFT data collection (September 2015 to January 2016).

We will purposively select 4 states and the District of Columbia, selecting states to represent each of the 4 Census geographic regions. In the 4 states, we will purposively select a single metropolitan statistical area (MSA). We will then select 50 schools from each state’s purposively selected MSA and select 50 schools from the DC MSA for a total of 250 schools; with the goal of obtaining 11-12 participating schools in each of the 4 states and the DC MSA. Some of the benefits of using MSAs over counties include:

(1) In states where districts and counties are synonymous, one doesn’t lose the entire area if a district refuses to participate;

(2) MSAs are better able to cover all of the urbanicity types; and, like counties,

(3) MSAs involve a reasonable travel range for field staff which leads to reduced collection costs due to the clustering of schools within defined geographic areas.

The 250 schools selected for the IVFT will be recruited at the same time. A selection of 58 schools from among those that agree to participate will be included in the IVFT. During this recruitment effort, participating schools will be tracked to ensure heterogeneity of demographics such as state and urbanicity. If, for example, a sufficient number of suburban schools from a particular state agree to participate, recruitment efforts with the remaining schools of that type will cease but will continue with the remaining schools in the state that are urban and rural.

If a selected school offers grade 5, the goal will be to assess all of the 5th-graders within the school. If the participating schools do not provide sufficient numbers of 5th-grade students to meet the sample targets, we will supplement the initial school sample with schools serving 5th-grade students. We will first seek schools with 5th grades in close proximity to other schools participating in MGLS. If additional schools need to be recruited to obtain the 5th-grade targets, they will be selected for cost effectiveness of field administration.

Also, if the schools do not provide sufficient numbers of students within the three focal disability categories to meet the sample yield targets, we will supplement participating schools with a sample of special schools where more than 80 percent of the students have been identified as having an IEP. We will initially seek to identify these schools in or near districts that already contain schools that have agreed to participate. If we need to recruit additional special schools, we will choose schools that allow the most cost effective field administration, given the geographic deployment of the field assessment staff.

If it is determined that additional schools, beyond the 58 Tier 1 schools, are needed to achieve the desired student yield within each of the subgroups, additional schools (Tier 2 schools) will be recruited to participate in the student component of the study. For the purpose of the IVFT, collecting data from school staff and parents in the 58 Tier 1 participating sample schools should be sufficient to inform the operational field test and main study questionnaire testing. Thus, only students will be assessed in Tier 2 schools to achieve the desired yield targets.

The opportunity for their students to participate in field-testing assessments for a national study is sometimes of considerable interest to school officials, and securing such “as needed” volunteer schools will safeguard the success of the IVFT. For these as-needed volunteer schools, depending on the school configurations and needs of the IVFT, participation may also be restricted to a subset of grades (e.g., one school may volunteer to have only their 5th-graders participate and another school may ask that only 8th-graders be included).

B.2 Procedures for the Collection of Information

School Recruitment Approach

Gaining schools’ cooperation in voluntary research is increasingly challenging. For example, in 1998–99 the Early Childhood Longitudinal Study had a weighted school-level response rate of 74 percent,¹ whereas 12 years later, the complementary ECLS-K:2011 study had a weighted school-level response rate of 63 percent.² Additionally, there is evidence that response rates may be lower for schools that serve older students, as in the High School Longitudinal Study of 2009, which had a weighted school-level response rate of 56 percent.³ Therefore, effective strategies for gaining the cooperation of schools are of paramount importance. Schools will be recruited both directly and at the district level.

Direct School Recruitment. In the interest of time, a sample of schools that do not require district level approval will be identified for direct recruitment. As part of the school recruitment, we will send an advance notification letter to their state (appendix B) and district (appendix C) education agencies, and then a trained school recruiter will initiate contact with the school. Schools will receive an attractive, introductory information packet via FedEx that includes a cover letter (appendix D), a colorful recruitment-oriented brochure (appendix H), and a sheet of Frequently Asked Questions (FAQs) about the study (appendix I) with links for accessing the MGLS:2017 recruitment website.

Within 3 business days of the information packet delivery (confirmed via FedEx), a school recruiter will follow up with a phone call to secure the school’s cooperation and answer any questions the school may have. During this call, the recruiter will establish who from the school’s staff will serve as the school coordinator for the study. The MGLS:2017 study team will then work with the school coordinator to schedule IVFT activities at the school, including gathering student rosters, distributing consent materials to parents of sample students, and arranging the onsite assessments. In early communications, the recruiter will also gather information about what type of parental consent procedures need to be followed at the school; any requirements for collecting data on the IEP status of students and student-teacher rosters; hours of operation, including early dismissal days, school closures/vacations, and dates for standardized testing; and any other considerations that may impact scheduling student assessments (e.g., planned construction periods, school reconfiguration, or planned changes in leadership). Participation refusals will be referred to an MGLS:2017 data collection task leader and examined for possible approaches to refusal conversion.

District-Level Recruitment. In addition to school recruitment, some districts may be asked for student rosters to enable MGLS staff to recruit enough students in the three focal disability categories, or they may be asked to endorse the study to aid the study’s school recruitment efforts. Generally, for district-level recruitment, a similar protocol will be followed as with direct school recruitment. Districts will receive an introductory information packet, including a cover letter (appendix C), brochure (appendix H), and FAQs (appendix I). Three days after mail delivery of the packet, a recruiter will make a call to secure the district’s cooperation, answer any questions the superintendent or other district staff may have, review the list of schools in the district serving grades 6–8, confirm key information about the schools (e.g., grades served, size of enrollment, enrollment of students with disabilities), and discuss obtaining students’ IEP information that is necessary for drawing the IVFT student sample. Information collected during this call will be used to confirm which schools in the district are eligible for participation in the study, and to obtain contact and other information helpful in school recruitment.

Once district cooperation has been obtained and the school sample has been selected, we will complete any required additional district-level paperwork, such as research applications. Recruitment of schools in these districts will follow the same protocol as outlined above under “Direct School Recruitment”, with the additional inclusion in the school introductory information packet of the district’s letter of approval and support. Participation refusals by districts or individual schools will be referred to an MGLS:2017 data collection task leader and examined for possible approaches to refusal conversion.

Private and Charter School Recruitment. If a private or charter school selected for the field test operates under a higher level governing body such as a diocese, a consortium of private schools, or a charter school district, we will use the district-level recruitment approach with the appropriate higher level governing body. If a private or charter school selected for the field test does not have a higher level governing body, we will use the direct school recruitment approach outlined above.

Student Rosters Collection. Once a school or district has agreed to participate, the MGLS contractor, RTI, will gather student rosters from the district or with the assistance of the school coordinator from the school. A complete roster of all students eligible for sampling will be requested, including key student characteristics, such as: name; ID number; date of birth; grade level; gender; race/ethnicity; and IEP status with disability code(s), when applicable. Based on this information the student sample will be drawn. As part of the roster collection, RTI will also request from the school coordinator or designated district personnel the following information for each student eligible for sampling: student’s parent and/or guardian contact information (e.g., mailing address; landline phone number; cell phone number; e-mail address); student’s math teacher; and student’s special education teacher, when applicable. Schools and districts usually find it easiest, and therefore most efficient, to supply all of the desired information one time for all of their students. However, should it be problematic for any school or district to provide the parent and teacher information on the complete roster, RTI will gather that information as a second step for the sampled students only. If the school and/or district is unwilling to provide parent contact information, RTI will work with the school and/or district to determine the best way to contact parents (e.g., the school coordinator or designated district personnel would facilitate contacting parents and/or mail the required materials to parents using the contact information they have on file).

Schools and districts will be provided with a template and secure transfer options to deliver the rosters (appendices S and T). The data quality of the student rosters will be then evaluated by:

reviewing and assessing the quality and robustness of student and parent information available at each school, including contact information for parents;
reviewing and assessing the quality of the data on student-teacher linkages;
reviewing and assessing the quality of the data on IEP status;
addressing any incompleteness or irregularities in the roster file;
requesting additional information as needed from the school coordinator or designated district personnel; and
(re)verifying that the sampled students are currently in attendance in the school.

Parent Recruitment

Information about schools’ procedures for obtaining consent for students to participate in the study will have been gathered during school recruitment. Schools generally require one of two types of consent: implicit or explicit. Both types of consent require that parents be notified that their children have been selected for the study. With implicit consent, the school does not require verbal or written consent for a student to participate in the study – parents are asked only to notify the appropriate person if they do not want their child to participate (appendix F). With explicit consent, children may participate only if their parents provide written or oral consent for their children to do so (appendix G). In the field test, proactive parent recruitment will be focused on maximizing the number of parents (1) returning signed explicit consent forms and (2) completing the parent survey.

After the student sample is drawn within a school, the initial communication with parents consisting of introductory and consent materials (Appendixes F-I) will be distributed to parents in a way each school believes to be most appropriate and effective (e.g., sending the materials home with students; the school or district sending the materials directly to parents; and/or trained MGLS recruitment staff contacting parents directly by mail, email, and/or phone). The initial materials will introduce the study, explain the study’s purpose and the importance of student and parent participation, describe what is involved in participation, and specify the consent procedure that is being used by their school. The materials will include a consent seeking letter to all parents plus a consent form where explicit consent is required (appendices F-G), a colorful recruitment-oriented brochure (appendix H), and a sheet of FAQs about the study (appendix I) with links for accessing the MGLS:2017 recruitment website. Additionally, in schools using explicit consent, the parental consent form for student’s participation, which will be included in the initial communication materials, will ask parents to provide their contact information (appendix G).

The IVFT will include a parent incentive experiment. All parents of sampled students within the same school will be assigned to the same incentive level. Within the schools selected for a parent incentive other than $0, the $20 or $40 incentive amount will be indicated in the letter that will accompany the initial parent contact materials and will be referenced on the explicit consent form (appendices F-G).

In schools that use an explicit consent procedure, the methods that will be used to follow up with parents who do not respond to the initial contact materials will depend on whether the school or district provided contact information for these parents and on the ways each school considers to be appropriate for contacting parents (either through the school’s staff or by MGLS trained staff directly). The non-response follow-up will focus on assuring that parents receive the introductory and consent materials, encourage them to sign and return the consent form, and answer any questions they may have. In schools that use an implicit consent procedure, no follow-up will be required given that absence of parents’ reply indicates consent.

If parents indicate that they would prefer not to participate at a rate that suggests refusal conversion is necessary for the success of the field test, trained recruiters may contact parents in an attempt to convert refusals by addressing any concerns parents may have over participation in the study. Depending on the mode of initial contact, parents may be contacted by phone, email, or a mailing sent home from RTI or from the school or district personnel. Refusal conversion experiences in the IVFT may inform parent recruitment materials and procedures that will be used in the Operational Field Test (OFT) and the main study.

B.3 Maximizing Participation

Studies increasingly experience challenges in obtaining the cooperation of districts and schools. Loss of instructional time, competing demands (such as district and state testing requirements), lack of teacher and parent support, and increased demands on principals impede gaining permission to conduct research in schools. MGLS:2017 recruitment teams will be trained to communicate clearly to districts, dioceses, private school organizations, schools, teachers, parents, and students the benefits of participating in the field test and what participation will require in terms of student and school personnel time. MGLS staff will utilize conferences to educate middle grades professionals about the study and its field tests and to increase MGLS name recognition.

As part of the strategy to maximize response rates among school districts and schools during the recruitment process, RTI has established partnerships with organizations such as the Association for Middle Level Education (AMLE) and the National Forum to Accelerate Middle-Grades Reform (the Forum). These organizations will actively promote the value of the study to their constituencies as will a number of middle-grades education researchers who will participate in the recruitment effort.

Representatives from these organizations have committed to provide outreach to sample schools via direct letters and emails, and to the middle grades community in general via information in newsletters and related communications. These communications will include information about the importance of the study, what study participation entails, and the benefits of the study to the middle grades community. Direct communications with the schools selected for the IVFT will let schools know that they may also be selected for the main study (the IVFT will take place during the 2015-16 school year, while the main study will take place two years later, during the 2017-18 school year).

Recruiters will be trained to address concerns that districts and schools may have about participation, while simultaneously communicating the value of the study and the school’s key role in developing instruments that ensure high-quality data focusing on middle-grade students. Early engagement of districts and school administrators will be important. The study will also offer monetary incentives to schools (see Part A), which have proven in other NCES studies to increase school participation rates.

Along with offering monetary incentives, our plan for maximizing district, school administrator, and parent engagement includes the following:

Experienced recruiters. The recruiting team will include staff with established records of successfully recruiting school districts and schools. To maximize district approval, senior staff will make the initial district telephone contacts. Their familiarity with the study and its future impact, as well as their experience in working with districts to overcome challenges to participation, will be crucial to obtaining district approval. Recruiters contacting schools will be equally adept at working with school administrators and providing solutions to the many obstacles associated with student assessments, including conflicts related to scheduling and assessment space, minimizing interruption to instructional time, and obtaining teacher and parent buy-in.
Persuasive written materials. Key to the plan for maximizing participation is developing informative materials and professional and persuasive requests for participation. The importance of the study will be reflected in the initial invitations from NCES (appendices C-G) sent with a comprehensive set of FAQs (appendix I) and a colorful recruitment-oriented brochure describing the study (appendix H). Reviewing these study materials should provide districts and school administrators with a good understanding of the study’s value, the importance of the field test, and the data collection activities required as part of the study. A full understanding of these factors will be important both to obtain cooperation and ensure that schools and districts accept the data collection requests that follow.
Persuasive electronically accessible materials. In addition to written materials, we will develop a recruitment-focused website which, drawing heavily on the written materials, will present clear and concise information about the study and convey the critical importance of participating in it. AMLE and the Forum will provide an outreach service, asking for support of the study, offering updates to their constituencies on the progress of the study, and making available information on recent articles and other material relevant to education in the middle grades.
Buy-in and support at each level. During district recruitment, the study team will seek not just permission to contact schools and obtain student rosters but also to obtain support from the district. This may take the form of approval of a research application and a letter from the district’s superintendent encouraging schools to participate. Active support from a higher governing body or organization, such as a district or a diocese, encourages cooperation of schools. Similarly, when principals are interested in the research activity, they are more likely to encourage teacher participation and provide an effective school coordinator.
Avoiding refusals. MGLS recruiters will work to avoid direct refusals by focusing on strategies to solve problems or meet obstacles to participation faced by district or school administrators. They will endeavor to keep the door open while providing additional information and seeking other sources of persuasion.

B.4 Purpose of Field Test and Data Uses

The IVFT is designed to obtain the cooperation of enough schools in order to sample and collect sufficient data from students, parents, teachers, and administrators to evaluate the psychometric properties of items and the predictive potential of scale variables. This will allow valid, reliable and useful assessment and survey instruments to be composed for the main study, with the OFT serving as a second opportunity to test all assessments and surveys and all procedures one year in advance of the main study. Additionally, as described in Part A, section A.9, the IVFT will include a school and a parent incentive experiment to determine whether different levels and/or types (in case of schools) of incentives can significantly improve participation rates.

B.5 Individuals Responsible for Study Design and Performance

The following individuals at NCES are responsible for the MGLS:2017: Carolyn Fidelman, Gail Mulligan, Chris Chapman, and Marilyn Seastrom. The following individuals at RTI are responsible for the study: Dan Pratt, Debbie Herget, Steven Ingels, and David Wilson, along with subcontractor staff: Sally Atkins-Burnett (Mathematica) and Michelle Najarian (ETS).

1 Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G., Hagedorn, M.C., Daly, P., and Najarian, M. (2001). Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K), User’s Manual for the ECLS-K Base Year Public-Use Data Files and Electronic Codebook (NCES 2001-029). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

2 Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G., Hagedorn, M.C., Daly, P., and Najarian, M. (2012). Early Childhood Longitudinal Study, Kindergarten Class of 2010–11 (ECLS-K:2011), User’s Manual for the ECLS-K:2011 Kindergarten Data File and Electronic Codebook (NCES 2013-061). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

3 Ingels, S.J., Pratt, D.J., Herget, D.R., Burns, L.J., Dever, J.A., Ottem, R., Rogers, J.E., Jin, Y., and Leinwand, S. (2011). High School Longitudinal Study of 2009 (HSLS:09). Base-Year Data File Documentation (NCES 2011-328). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	anon
File Modified	0000-00-00
File Created	2021-01-24

Part B MGLS 2017 IV Field Test 2016 Recruitment