Part B ICILS 2023 Main Study Data Collection

The respondent universe for the ICILS main study is all students enrolled in grade 8, provided that the mean age at the time of testing is at least 13.5 years of age. The school target population include schools offering eighth-grade instruction in the winter/spring of 2023 in the 50 states and the District of Columbia. These schools include:

Public schools, including charter and magnet schools;
Private schools; and
Bureau of Indian Education schools.

Although the following schools are eligible for ICILS 2023 and could be considered for the main study, we plan to exclude them from the main study. However, we note that changes in the number of virtual schools, the number of students attending virtual schools, and/or school classifications in light of increased remote learning during the coronavirus (COVID-19) pandemic may require adjustment of the virtual schools’ exclusion. Currently, our plan models that planned for TIMSS 2023 and excludes:

Special education schools,
Virtual schools, and
Schools that include temporary housing like correctional facilities and hospitals.

The teacher target population in ICILS consists of all teachers that are teaching regular school subjects to 8th grade students (regardless of the subject or the number of hours taught) during the ICILS testing period and since the beginning of the school year. A sample of 186 schools will be selected for the main study, with the goal of obtaining participation from a minimum of 150 schools.

In each sampled school, one or two classes of students will be selected. A minimum of 15 teachers will be selected. If fewer than 15 eligible teachers can be found in a school, they will all be included in the survey. Also, if the number of eligible teachers is greater than 15, but less or equal to 20, all teachers will be selected in order to prevent the situation that a small number of teachers are the only ones not being surveyed.

School principals and ICT coordinators of the sampled schools will also be asked to complete a school questionnaire.

The student target population for the main study of ICILS 2023 is eighth-grade students enrolled in winter/spring of 2023 in the United States (50 states and the District of Columbia) in a school in the school target population.

B.2 Statistical Methodology

The school sample design for the main study uses probability sampling to select a sample of schools for grade 8. The school sample will represent all schools, with grade 8, in the entire United States and D.C. The main study for ICILS will take place in the spring of 2023.

The number of overlapping schools between ICILS and other studies will be kept to a minimum. The Trends in International Mathematics and Science Study (TIMSS) 2023 main data collection, for students in grades 4 and 8, will take place at the same time as ICILS. Overlap control procedures in studies such as this, where stratified probability proportional to size samples of schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional probability of selection of a given school for one study, depending upon its selection probability for a second study, and whether or not it was selected for that study. This approach was first documented in a survey sampling application by Keyfitz (1951)¹. The principles involved can be extended to more than two studies simultaneously, and a procedure for doing this is described by Chowdhury et al. (2000)².

The sample size for the ICILS main study will be 186 schools, with two replacement schools available for each of the 186 schools. The sampling frame of grade 8 schools will be obtained from NCES’ 2019-2020 Common Core of Data (CCD) and preliminary 2019-2020 Private School Universe Survey (PSS) files, restricted to schools having grade 8, respectively, and limited to schools in the 50 United States and the District of Columbia.

The school sample will be explicitly and implicitly stratified according to school characteristics. Explicit strata will be created by cross-classifying school type (public/private), Census region (Northeast, Midwest, South, and West), and poverty status (High/Low, as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools. Within each explicit stratum, schools will be hierarchically sorted by locale (city, suburb, town, rural), percent non-white (less than 15 percent, greater than or equal to 15 percent), State, and enrollment³ in the respective grade. Hierarchical sorting will result in samples within explicit strata being distributed approximately proportionately across the groups defined by the implicit stratification variables.

Schools will be selected with probability proportional to enrollment in grade 8 using systematic probability proportional to size sampling. Because the number of classes in a given grade at a school is approximately proportional to the number of students enrolled in that grade, and because one or two classrooms will be sampled from participating schools, the use of a probability proportional to enrollment sample design ensures that all students have an approximately equal chance of selection, since up to two classes will be selected from each school regardless of the size of the school.

Student sampling will be accomplished by selecting one or two classes per school. Participating schools will be asked to provide a list of classes. Students will be selected for participation by drawing a random sample of up to two classes in each school using software provided by the international ICILS study coordinator. Smaller classes may be combined to form ‘pseudoclasses’ for sampling. After sampling, schools will provide the list of students in the sampled classes. All selected students will be asked to participate in ICILS. We estimate that on average 45 students will be selected from each school, and up to 10 percent of those students will be ineligible or excluded⁴ for the main study, yielding approximately 10,000 eighth grade students.

Class and student lists will be gathered from participating schools electronically using a secure electronic filing process (as explained in Part A). Electronic filing provides advantageous features such as efficiency and data quality checks. Schools will access the electronic filing system through a web site.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

We will conduct a nonresponse bias analysis pursuant to NCES statistical standards in order to determine where and how respondents and nonrespondents differ on available characteristics. Sampling errors, that account for unequal weighting, stratification, and probability sampling will be calculated for a selection of key estimates. For more details about our planned response, please see our Non-Response Bias Analysis Plan in Appendix C.

B.3 Maximizing Response Rates

Gaining cooperation from school districts and schools is paramount to the success of ICILS 2023, and also the most significant challenge of the study. In 2018, student participation was challenging for ICILS than other international assessments because ICILS required a random student sample within schools rather than a selection of classes. For 2023, ICILS has followed the Trends in International Mathematics and Science Study (TIMSS) and changed the student sampling method to select intact classes. Historically, TIMSS had overall student participation rates above 90 percent (see Table 1). In addition, it is important to U.S. ICILS that students are engaged and try to do their best on the assessment.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
Year	Grade	Before Replacement	After Replacement	Overall Student Participation Rate
2019	4	76	88	96
2019	8	72	85	94
2015	4	79	85	96
2015	8	78	84	94
2011	4	79	84	95
2011	8	87	87	94
2007	4	70	89	95
2007	8	68	83	93
2003	4	70	82	95
2003	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to maximizing school recruitment is to:

Begin recruitment activities as early as possible;
Engage stakeholders at all levels throughout the cycle of ICILS;
Obtain endorsements about the value of ICILS from relevant national organizations, state and regional agencies;
Engage with the broad school community (i.e., not sampled schools) through involvement at small-scale school leader conferences and events, to provide an opportunity to test-drive our anticipated recruitment messages, build relationships with influential school-level leaders and highlight anticipated benefits of participation to build hope and excitement among schools that they may be selected into the study;
Inform Chief State Officers and state assessment directors about the sample of schools in their state;
Use the assistance of NAEP State Coordinators to recruit districts and schools, providing key state agency involvement in recruitment;
Send letters and informational materials to schools and districts. These letters will be customized by type of school. In addition, alternative versions are provided in Appendix A to account for school composition during recruitment, as additional language will be added for schools conducting learning virtually;
Train experienced NAEP State Coordinators about ICILS;
Follow-up mailings with telephone calls and emails to explain the study and school involvement, including placing the ICILS assessment date on school calendars;
Maintain continued contact until schools have built a relationship with the ICILS team and fully understand ICILS;
Use monetary school incentives of at least $200 and up to $800 for select schools (all school incentives will be $200, and graduated incentives will be introduced for target schools, per description in Part A.9);
Offer salient, interactive, topical webinars to participating district and school staff;
Provide school reports augmented with survey response data;

Produce and distribute a newsletter compiling relevant elementary and middle-school math and science research reports and practice guides from federal Regional Education Laboratories, Comprehensive Centers, and the What Works Clearinghouse for all levels;

Provide study updates for all levels on key study milestones and releases; and
Make in-person visits to some districts and schools, as necessary.

Our approach to maximizing student recruitment is to:

Encourage schools to use implied permission forms or notification letters. Written permission will be collected if required by the school district or school;
Remind school coordinators to send parental permission forms home to parents 4-6 weeks ahead of the ICILS sessions to allow ample time for parents to grant permission;
Offer participating students a small token of appreciation valued at approximately $4, for example ear buds, a digital watch, or a pair of sunglasses;
Provide students with a certificate with their name thanking them for participating and representing the United States in ICILS;
If permitted by the school, certificate will also serve as a certificate of service and community service hours, as permitted by the school;
Remind school coordinators to encourage student participation;
Ask schools to hang colorful, engaging posters in the school announcing participation;
When feasible, have the test administrator (TA) speak to student prior to the scheduled session day to encourage participation.

Our approach to maximizing teacher recruitment is to:

Send letters and materials reinforcing the importance of participation to teachers;
Follow up with prompting phone calls from RTI’s recruitment experts, who have already established relationships with the school during the school recruitment phase;
Offer a $25 incentive for participation;
Have the TA speak to teachers in person on the day of the student session, or, if teachers are not available, have TA leave personalized postcard reminders in teacher mailboxes.

B.4 Purpose of the Main Study and Data Uses

The goals of the ICILS main study are to (1) provide trend information about student achievement in mathematics and science to inform education policy discussions and identify existing educational inequalities; (2) provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance.

Data compiled and collected from ICILS 2023 will allow for evidence-based decisions to be made for educational improvement. These high-quality, internationally comparative trend data are key to informing education policy discussions.

B.5 Individuals Consulted on Study Design

Overall direction for ICILS is provided by Linda Hamilton, National Research Coordinator, National Center for Education Statistics (NCES), U.S. Department of Education, in consultation with a number of NCES statistical staff.

IEA is responsible for the statistical design of ICILS.

RTI International is the contractor responsible for sampling, data collection, and data analysis:

Debbie Herget, Project Director, RTI International (919-485-7793);
Ben Dalton, Associate Project Director, RTI International (919-541-7228); and
David Wilson, Senior Statistician, RTI International (919-541-6990).

Analysis and reporting will be performed by:

National Center for Education Statistics, U.S. Department of Education;
ICILS International Study Center, IEA;
RTI International; and
AnLar, under contract to RTI International.

1 Keyfitz, N. (1951). Sampling with Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/ proceedings/papers/2000_025.pdf.

3 Enrollment less than 5 will be set to 5 for purposes of implicit stratification and sampling.

4 Students are considered ineligible if they are enrolled in the classroom but not in the target grade (grade 8) or if a student transferred from the school or class between the time that the student roster was prepared and the scheduled session at the school. Students are excluded if they meet the internationally defined criteria as having a functional disability, an intellectual disability, or if they are a non-native language speaker and unable to read or speak the language of the assessment. Students with functional or intellectual disabilities or have at least one year of English instruction or are able to overcome the language barrier would be included in the testing.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Millar, Kimrey
File Modified	0000-00-00
File Created	2022-01-20

Part B ICILS 2023 Main Study Data Collection