International Early Learning Study (IELS) 2018
Field Test Data Collection and Main Study Recruitment
OMB# 1850-0936 v.3
Supporting Statement Part B
National Center for Education Statistics (NCES)
U.S. Department of Education
Institute of Education Sciences
Washington, DC
July 2017
TABLE OF CONTENTS
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1 Respondent Universe 2
B.2 Procedures for the Collection of Information 3
B.2.a Respondent Recruitment 3
B.2.b Data Collection 6
B.3 Maximizing Response Rates 8
B.4 Purpose of Field Test and Data Uses 9
B.5 Individuals Consulted on Study Design 9
COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Part B of this submission presents information on the collection of information employing statistical methods for all aspects of the International Early Learning Study (IELS) field test in 2017, recruiting for the IELS 2018 main study, and includes the overarching plan for the IELS 2018 main study data collection. Any revisions will be submitted to OMB for approval as a change request as soon as they become available.
The IELS assesses children as they transition to primary school. For international comparability, the U.S. study will focus on five-year-old students in public and private schools that offer kindergarten. The sample of age-eligible students in a given school will be drawn from across all grades in the school, not just kindergarten. In particular, a large proportion of schools that offer kindergarten also offer pre-kindergarten, and it is likely that some prekindergarten students will meet the age definition. These students will be eligible for inclusion. Similarly, there may be the occasional first grade student who meets the age definition, and thus will be included in the list for sampling. The expectation is that children of this age will be assessed near the beginning of the school year. This is an important transition point for U.S. children, and it is also the point at which it is feasible to reach at least 95 percent of U.S. children through a school-based sampling approach. The universe for the selection of schools for IELS is public and private schools with kindergarten, in all states of the 50 United States and the District of Columbia. Within sampled schools, students will be selected for participation by drawing a random sample among children that are 5-years-old during the assessment period.
Schools
Field Test
International standards do not require a formal probability sample of schools for the IELS field test. It is sufficient that the samples of schools be representative of a broad range of schools from across the U.S. Schools must be officially registered institutions that provide services for children of the target age for at least 2 hours per day and 100 days a year. The U.S. expects that 30 schools will participate in the field test.
For the IELS field test, schools will be selected for recruitment from the most recent available universe files, the 2015-16 Common Core of Data (CCD), to develop the public school list, and from the 2012-13 Private School Universe Survey (PSS) to develop the private school list. The less restrictive sampling requirements of the IELS field test mean that schools that require extensive research applications may not be included.
To obtain a school sample that is broadly representative of schools across the U.S., we will target a convenience sample of schools that offer kindergarten. We will use the sample stratification characteristics used in other studies of this type (e.g., PISA) including census region, locality (city/urban fringe/town/rural MSA), school type (public/private), grade span, and minority enrollment.
Main Study
The main study will be based on a formal probability sample designed to yield a representative sample of 3,000 children age 5 enrolled in schools that contain kindergarten. The U.S. expects a sample of 200 schools.
Schools will be selected for recruitment from the most recent available universe files, the 2015-16 Common Core of Data (CCD), to develop the public school list, and from the 2015-16 Private School Universe Survey (PSS) to develop the private school list. Schools under the jurisdiction of the Department of Defense (DoD) and the Bureau of Indian Education (BIE) will also be included in the frame.
Stratification for the main study sample will be both explicit and implicit. Public and private schools will constitute distinct sampling strata. Further stratification includes geographic region (northeast, mid-west, south, west), locale (city, suburb, town, rural), and socio-economic status as measured by the percent of students having free or reduced-price lunch.
Students
Field Test
The OECD IELS field test requirements call for a minimum student sample of 450 students, and we estimate that approximately 476 will participate. In each school in the field test, 19 students in the target population will be randomly sampled from a list of all target-age students, provided by the school in the field test, with the expectation that about 16 will participate. Depending on each school’s policy, either active (explicit) or passive (implicit) parental consent materials will be distributed to the sampled students within each school.
Main Study
The OECD IELS main study requirements call for a minimum student sample of 3,000 students, and we estimate that approximately 3,060 target-age children will participate. In each school in the field test, 19 students in the target population will be randomly sampled from a list of all target-age students provided by the school in the main study, with the expectation that about 16 will participate. Depending on each school’s policy, either active (explicit) or passive (implicit) parental consent materials will be distributed to the sampled students within each school.
This section presents information on the recruitment and data collection procedures for both the IELS field test and the main study. Gaining schools’ and students’ cooperation in voluntary research is increasingly challenging and employing effective strategies for gaining the cooperation of schools are especially of paramount importance. Main study states, districts, and private schools will be recruited beginning in the fall 2017. The recruitment effort will run concurrently with the field test district and school recruitment. Main study school recruitment will be conducted in winter and spring 2018.
The field test will be conducted in November through December 2017, while the main study will be conducted in October through November 2018.
B.2.a Respondent Recruitment
Based on recruitment knowledge gained in the ECLS-K:2011 and other NCES studies, states and districts should be notified about the study and persuaded to allow project recruiters to contact the sampled schools in their jurisdictions. School staff often want to be sure district cooperation is in place before agreeing to participate in the study.
Our proposed approach for state, district, and school recruitment is as follows. See Appendix A and B for all of the respondent materials referenced below.
State Recruitment. In fall 2017, state education agencies (SEAs) in states that contain sampled main study schools will be mailed a package that includes the state letter and IELS brochure. We will work with the NAEP State Coordinators (NSCs) in the state recruitment effort. Some NSCs may wish to send the state letter themselves, as well as follow-up to answer any questions, otherwise Westat school recruiters will mail the state package. Note these packages will be sent to SEAs for the main study only. Because the field test will use a purposive sample, states with especially engaged NSCs that will urge cooperation were selected and thus, such a package is not needed.
District Recruitment. In fall 2017, an advance package to field test and main study district superintendents will be mailed. The package will contain an introductory letter, including a list of sampled schools in their jurisdiction, and the IELS brochure. We will stagger the mailings to districts in those states that are also part of the field test. The district mailings will come from the NSCs or Westat, depending on NSCs’ preference. Shortly after the mailing, the district superintendent will be contacted by phone to ensure they received the package, answer any questions they may have, review the list of schools sampled in the district, confirm key information about the schools (e.g., kindergarten present, size of enrollment, etc.) and get permission to contact the schools in their districts. Any requirements regarding parent consent will also be identified.
Information collected during this call will be used to confirm which schools in the district are eligible for participation in the study, and to obtain contact and other information helpful in school recruitment.
The study staff will be prepared to respond to requirements such as research applications or meetings to provide more information about the study. If a district chooses not to participate, the recruiter will document all concerns listed by the district so that a strategy can be formulated for refusal conversion attempts.
Based on experience from previous NCES studies, some NSCs will want to talk to district staff themselves, while others will mail the package but do no further recruitment, and still others will not want to be involved in district recruitment at all. In cases where the NSCs do not wish to follow-up, Westat’s recruiters will be prepared to work directly with the districts.
Some districts will be designated as “special handling districts”, based on prior recruitment experience on a variety of NCES studies. Contacting special handling districts will begin with updating district information based on what can be gleaned from online sources, followed by calls to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry will also be made about the amount of time the districts spend reviewing similar research applications. To allow sufficient time for special handling districts’ review processes, this operation will begin upon receiving OMB’s approval, and will continue until we receive a final response (approval or denial of request) from each school, up until data collection.
School Recruitment. Once the district has agreed that their sampled schools may be contacted, the main study private school and field test public and private school mailings will be triggered. These mailings, which will be sent to field test schools in fall 2017, will contain an introductory letter, the IELS brochure, and school administrator Frequently Asked Questions (FAQs). Shortly after the mailing, the NSCs or Westat recruiters will phone the school administrator to discuss the study and gain cooperation, as well as to assign a school coordinator (SC) who will work with the IELS field staff to manage the data collection in the school. The school coordinator acts as the liaison between study staff and their school. Note that this mailing and follow-up effort will occur in early 2018 for main study public schools.
School Coordinator Contact. Shortly after permission has been granted from the school administrator, Westat team leaders will call the SC to discuss the IELS activities at the school, including gathering student rosters, distributing consent materials to parents of sample students and setting an assessment date. The team leaders will also gather information about what type of parental consent procedures need to be followed at the school; hours of operation, including early dismissal days, school closures/vacations, and dates for standardized testing; and any other considerations that may impact scheduling student assessments (e.g., planned construction periods, school reconfiguration, or planned changes in leadership). They will review the MyIELS website and how it will be used throughout the field period. If the SC would like more information about IELS, he or she will be directed to the website or, if preferred, receive a mailing of the school administrator FAQs and the study brochure. See the next section for a description of the SC’s responsibilities during the field period.
Westat will also email the school coordinator MyIELS website registration information. This MyIELS website will contain the IELS respondent materials, as well as e-filing instructions and templates, and other assessment logistical information. Each school will receive a unique ID that will support multiple schools users. Each user must, however, provide their own contact information and setup their own unique account. This registration process has been used across all of NCES’ international studies, and also in NAEP.
The SC contact will occur immediately after school recruitment. For the field test schools and the main study private schools, this will occur in fall 2017, while for the main study public schools, this will occur in early 2018. Because a lag occurs between the first SC contact and the main study data collection activities, an email will be sent to the main study SC during summer 2018 to remind them of the study and the upcoming tasks to be completed prior to the assessments. This email is not needed for the field test, as the first contact with the SCs occurs shortly before e-filing and the assessment activities.
Parent and Teacher Recruitment. After the student sample has been selected, parents and teachers will be recruited and asked to complete online surveys. Teacher names and email addresses will have been collected during the sampling phase, as the international sampling software requires schools to provide a roster of teachers as well as their students. Collecting both pieces of information at the same time allows for each student to be linked directly to his/her teacher. Edit checks will be run during sampling to ensure that all students have been linked to a teacher and that all teachers have students linked to them. If unlinked students or teachers are found, field staff will clarify and correct the issue with the SC.
After sampling has been completed and the correct teacher names and email addresses have been collected, the Westat team leaders will talk with the SC about distributing study information to the teacher respondents. SCs will be mailed IELS brochures and teacher FAQs and asked to distribute them to the appropriate teachers. The SC will serve as the study’s main point of contact for teachers, and will answer any questions from them about the study and assessment logistics. They will also inform the teachers of the assessment date. Shortly after sampling, Westat will email the teachers a direct, unique link to the teacher questionnaire.
In addition, because the sampled children themselves will not be directly recruited, their parents will also be asked to give permission for their participation. Information about schools’ procedures for obtaining consent for students to participate in the study will have been gathered during school recruitment. Schools generally require one of two types of consent: implicit or explicit. Both types of consent require that parents be notified that their children have been selected for the study. With implicit consent, the school does not require verbal or written consent for a student to participate in the study – parents are asked only to notify the appropriate person if they do not want their child to participate. With explicit consent, children may participate only if their parents provide written or oral consent for their children to do so. In a few cases, schools may choose simply to notify the parents of the study and their child’s participation.
Proactive parent recruitment will be focused on maximizing the number of parents who: (1) return signed explicit consent forms and (2) complete the parent questionnaire. While talking with the SCs, the team leaders will discuss procedures for obtaining contact information for the sampled children’s parents. This information is necessary to follow-up on parent consent forms and questionnaire response. Ideally, the school will provide parents’ email address, telephone/cell phone number, and address. The email address will be used to invite parents to register on the MyIELS website, as well as to follow-up on questionnaire nonresponse. The telephone number will also be used for consent follow-up and questionnaire nonresponse. The address will be used to mail a thank-you letter and the incentive once a parent or legal guardian completes the parent questionniare. SCs will have the option of uploading the parent contact information as an Excel file, entering the information into MyIELS, or telling the information to the team leader over the phone, and she will then enter it into MyIELS. This method was used successfully in ECLS.
We recognize that some SCs may be reluctant to provide parental contact information. The team leaders will work with the schools to identify the best method for contacting parents about the study and providing them with the consent forms and the link to the online questionnaire. Informational parent packets will be available, consisting of a letter specific to the type of consent that will be used, the appropriate consent form, the study brochure, parent FAQs, and a postcard containing MyIELS registration details. These materials will be distributed to parents in the way each school believes to be most appropriate and effective (e.g., sending the materials home with students; the school or district sending the materials directly to parents; and/or field staff contacting parents directly by mail, email, and/or phone). The approach to following up on consent will also be tailored to the school’s preferences. In most cases, we anticipate the SC will be the main contact for the parents and will follow-up on missing consent forms and questionnaires. Westat field staff will be available to assist in whatever way is needed.
Role of the MyIELS Website. We plan to use the MyIELS website to stay in touch with school administrators, SCs, and parents over the course of the school recruitment period. Parents and teachers are not required to register individually for the website, though parents will be encouraged to visit the website, update their contact information, provide consent, and be taken via hyperlink to the questionnaire site. Alternatively, they may go directly to the questionnaire site and complete their parent questionnaire. The goal is to make it as frictionless as possible to get parents and teachers into the system and capture their responses. While we typically do not drive the district to the website, provisions are in place for an account to be created for them if requested.
B.2.b IELS Data Collection
Data collection will occur in November-December 2017 for the field test and in October-November 2018 for the main study. Prior to each data collection, Westat field staff will complete various pre-assessment activities.
Pre-assessment Activities
In the fall of 2017 Westat field staff will contact the field test SCs to discuss the tasks the SC will be responsible for and review assessment day logistics. E-filing procedures, distribution of parent consent forms, and outreach to teachers will also be reviewed. There will be a short window in the field test from when the school will be recruited until assessment day. Thus field staff will reach out to SCs immediately after the school agrees to participate.
The main study timeline is different in that there will be a longer period of time from when the school is recruited until the assessment date. Main study schools will be recruited in early 2018, and Westat school recruiters will contact the SCs at that time to introduce the study, briefly review their responsibilities, and set an assessment date. However, most of the pre-assessment activities – e-filing, distribution of parent consent materials, outreach to teachers, etc. – will not be completed until the fall of 2018. These activities must occur after the start of the 2018-2019 school year in order to have the appropriate-aged children participate in the study.
Although the timeline for completing the pre-assessment activities is different, the activities themselves are the same for the field test and the main study. Westat field staff will discuss the SC’s role in the study, and ask the SC to do the following:
gather a roster of children born within the appropriate timeline, along with other needed information such as the children’s grade, gender, and teacher, and e-file it using the appropriate template;
after the sample is drawn, distribute parent consent materials;
arrange for an assessment date and location;
notify teachers of the study and distribute study materials;
welcome the assessment team on assessment day; and
work with the teachers and assessment team to help the assessment days run smoothly.
Full details about the SCs role and responsibilities will be posted to MyIELS:
Set an assessment date (made up of two days), making sure to avoid conflicts with any special events in the school’s calendar, such as a field trip or school holiday.
Arrange for an assessment location, with the goal to identify locations that: provide as little distraction as possible, protect the privacy of the children, and are as non-disruptive to the school routine as possible.
Discuss the sampling procedures with IELS staff and receive assistance as needed in the e-filing process.
Work with IELS staff to distribute and collect parent consent forms.
Ensure that teachers have been notified of the study and are able to complete the questionnaire. Work with IELS staff to follow-up on missing or incomplete questionnaires.
As part of the pre-assessment activities, Westat will email the SC instructions for e-filing, including preparing an electronic file of students and teachers and submitting it using a standardized template. IELS help desk and field staff are resources for the SC to use while completing the sampling process. Once the sample has been drawn, the SC will be able to view the selected students on the MyIELS website. The field staff can then work with the SC to distribute parent consent and notify the appropriate teachers.
Throughout these pre-assessment activities, IELS staff will establish a positive and cooperative working relationship with school personnel and the school community. Westat’s field staff will communicate with the SC using email and phone. We expect that completion of the pre-assessment activities will necessitate multiple contacts.
Data Collection
Child Assessment. The direct child assessment will be a one-on-one untimed assessment composed of tasks that assess a variety of skills, with electronic data capture via tablet (about 60 minutes per child, conducted over a two-day period). Direct assessment items to be administered in the IELS field test will assess the following domains: language/emergent literacy, mathematics/numeracy, executive function/self-regulation, and social emotional skills (prosocial behavior, empathy, and trust). Results of the field test will be used to determine which items will be included in the main study. The direct assessment will be administered in English. For the main study, NCES may also choose to offer one or more domains in Spanish pending expert input and characteristics of the sample, in which case the Spanish versions will be submitted to OMB as a change request as soon as they become available.
Generally, the assessment team that visits the school will include a Westat team leader and three assessors. All of the field staff will be well-trained and experienced working with children of this age. The assessment team will arrive at the school on the appointed first day of assessments and, following the school’s required check-in procedures, immediately contact the SC. The team leader will introduce the assessors to the SC. The procedures to be used during the on-site data collection period will be discussed with the SC to ensure there is a common understanding of those procedures. The team leader will also confirm that all sampled children are still enrolled in the school as of the assessment day and determine which children are at school on the assessment day.
On the day of the IELS assessment, the team leader and assessors will be taken by school personnel to the assessment area(s), where they will remove potential distractions as much as possible and establish a comfortable environment for conducting the assessment. They will set up the assessment materials and log into the child assessment app on the tablets that they will bring with them. All field staff will be provided with backup batteries, cords, etc., to ensure that data collection activities are not disrupted by equipment problems.
Once the assessment areas have been set up and assessors are ready to begin work, the SC will introduce the IELS team members to the teacher(s) whose children will be assessed. The teacher(s), in turn, will introduce the assessors to the class. Assessors will then escort the sampled children to the assessment areas, one-by-one.
When the assessor and study child arrive at the assessment space, the assessor will introduce the child to the task and begin the practice items. After completing the cognitive assessments, the assessor will return the child to his/her classroom and the next sampled child will be assessed. These procedures will again be followed on day 2 of the data collection, with the same children completing the second part of their assessment.
It is expected that some children will be absent from school when the assessments are scheduled. Certain days throughout the field period will be designated as days on which some field staff can conduct make-up assessments. Attempts will be made to conduct a make-up assessment at some point during the field period for all children absent on their school’s assessment days.
Parent and Teacher Questionnaires. Contextual information will be collected from parents/guardians and teachers to gather information on the relation between children’s learning and development and on important demographic, social, economic, and education variables.
The parent questionnaire will ask for information about the child and the home, parents’ perspectives on their children social and emotional skills, the sampled child’s early childhood education and care participation, and home learning environment. The teacher questionnaire will ask teachers to provide information about their own background and education, and on each study child’s knowledge and skills. Both questionnaires will be available electronically and in hard copy. Draft field test questionnaires, which include the final international field test versions with proposed additional U.S. adaptation items, are included in Appendix C (revised July 2017). The final field test instruments, including all internationally approved U.S. adaptations, and including the Spanish-language version of the parent questionnaire, will be submitted to OMB as a change request upon approval of this request in August 2017.
As described above, parents and teachers will be given links to their questionnaires as part of their welcome materials. Parents will receive a postcard in their packet that includes their registration information for MyIELS. They will be directed to the website to register, provide consent electronically, and complete their questionnaire. If they do not wish to register and provide consent on MyIELS, they are also provided with a direct, unique link to the questionnaire. Teachers are emailed their link to the questionnaire and do not need to register on MyIELS unless they would like more information about the study.
Throughout the field test period, field staff and the SC will monitor the return of questionnaires, and, working in conjunction, will follow-up with non-responders as needed.
After Assessment Day. Once the data collection is completed, thank you letters and incentive checks will be distributed to responding parents, teachers, SCs, and school administrators. The letter will point respondents who wish to learn more about the study to the NCES IELS website.
Studies have increasingly experienced challenges in obtaining the cooperation of districts and schools. Loss of instructional time, competing demands (such as district and state testing requirements), lack of teacher and parent support, and increased demands on principals impede gaining permission to conduct research in schools. The IELS recruitment teams will be trained to communicate clearly to districts, dioceses, private school organizations, schools, teachers, and parents the benefits of participating in the IELS field test or main study, as appropriate, and what participation will require in terms of student and school personnel time. Recruiters will be trained to address concerns that districts and schools may have about participation, while simultaneously communicating the value of the study and the school’s key role in developing instruments that ensure high-quality data focusing on young kindergarten students.
Our approach to maximizing school and student response rates in the main study includes the following:
Assigning personal recruiters for specific schools;
Using experienced recruiters;
Developing persuasive written materials;
Avoiding refusals by focusing on strategies to solve problems or meet obstacles to participation;
Incentives for schools, school coordinators, and teachers (see Section A9); and
Contact with schools and school coordinators at set intervals throughout the year preceding the assessment.
These approaches are based on recommendations from an NCES panel and experience with previous administrations of international assessments such as PISA and TIMSS.
Participation in the field test is an international requirement for participating in the IELS main study. The main focus of the field test is to collect enough assessment data to perform reliable tests of the items. However, during the field test, procedures for conducting the main study, including recruitment methods for obtaining school and student participation also will be evaluated. This information will be used to: (a) determine the final main study design, and (b) improve our recruiting strategies and materials for the main study.
Overall direction for the IELS is provided by Dr. Dana Kelly, Branch Chief for International Assessments, and Ms. Mary Coleman, Project Officer, at the National Center for Education Statistics (NCES), U.S. Department of Education.
Key international colleagues include the following:
Overall project direction: Dr. Sacha DeVelle, Australian Council for Educational Research;
Survey design: Dr. Wolfram Schultz, Australian Council for Educational Research; and
Study Operations: Ms. Julianne Henke, IEA Data Processing Center.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |