Trends in International Mathematics and Science Study (TIMSS 2023) Field Test Data Collection and Main Study Sampling, Recruitment, and Data Collection

Trends in International Mathematics and Science Study (TIMSS 2023)

Field Test Data Collection and

Main Study Sampling, Recruitment, and Data Collection

OMB #1850-0695 v.17

Appendix D

TIMSS 2023 Nonresponse Bias Analysis Plan

Submitted by

National Center for Education Statistics

U.S. Department of Education

August 2021

Outline of the TIMSS 2023 Non-response Bias Analyses

RTI is planning to conduct a non-response bias analyses (NRBA) for schools if the response rate for original schools is below 85 percent. Currently, it is assumed that the student response rate for TIMSS 2023 will be above 85 percent and thus, no NRBA at the student level will be required.

TIMSS 2023 School Non-Response Bias Analysis Outline

If needed, the NRBA will be conducted in the summer/fall of 2024 before the release of the TIMSS 2023 national report in December of 2024. A summary of the findings will be included in the technical appendix of the U.S. national report and the full NRBA will be included in the U.S. technical report.

1.        INTRODUCTION  

A school NRBA is intended to assess the degree to which the characteristics of participating and non-participating schools differ. Large differences between the characteristics of participating and non-participating schools may indicate that estimates based on survey responses from participating schools may be biased. The characteristics of participating and non-participating schools will be compared and the degree to which observed differences are reduced through application of school-level nonresponse adjustment will be examined.

2.        METHODOLOGY

The analysis will be conducted in three parts:

1. The distribution of the participating original school sample will be compared with that of the total eligible original school sample. The original sample is the sample before substitution. In each sample, schools will be weighted by their school base weights and their estimated grade 4 or 8 enrollment, excluding any non-response adjustment factor.

2. The distribution of the participating final sample, which includes the participating replacements for schools from the original sample that did not participate, will be compared to the total eligible final sample. The final sample is the sample after substitution. Again, school base weights and their estimated grade 4 or 8 enrollment will be used for both the eligible sample and the participating schools.

3. The same sets of schools will be compared as in the second analysis but, this time, when analyzing the participating schools alone, school nonresponse adjustments will be applied to the weights.

The following categorical variables will be available for all schools:

• School type—public or private;

• Locale—urban-centric locale code (city, suburb, town, rural);

• Four-level Census region (Northeast, Midwest, South, and West);

• Poverty level—for public schools, a high poverty school is defined as one in which 76 percent or more of the students are eligible for participation in the National School Lunch Program (NSLP), and a low poverty school is defined as one in which less than 76 percent are eligible; all private schools are treated as low poverty schools; and,

• Percentage non-white—a school with 15% or more non-white students is classified as having a high percentage of non-white students while a school with less than 15% non-white students is classified as having a low percentage of non-white students.

The following continuous variables will be available for all schools:

• Number of grade-eligible (grades 4 or 8) students enrolled;

• Total number of students;

• Percentage of students by race/ethnicity (White, non-Hispanic, Black, non-Hispanic, Hispanic, Asian, American Indian or Alaska Native, Hawaiian/Pacific Islander and two or more races).

Two forms of analysis will be undertaken:

• A test of the independence of each school characteristic and participation status, and

• A generalized exponential regression in which the conditional independence of these school characteristics as predictors of participation will be examined.

For categorical variables, the distribution of frame characteristics for participants will be compared with the distribution for all eligible schools. The hypothesis of independence between the characteristic and participation status will be tested using an adjusted Wald-F statistic at the 5 percent level (SUDAAN 2012). For continuous variables, summary means will be calculated and the difference between means will be tested using a t test. In addition to these tests, generalized exponential regression models (including all characteristics) will be used to provide a multivariate analysis in which the conditional independence of these school characteristics as predictors of participation will be examined.

3.        RESULTS

For each categorical or continuous variable, a table will be shown giving the percentage (or mean) for the participating and eligible populations along with the bias, relative bias, and the p-value of the test. Text summaries of the results will also be provided. The generalized exponential regression results will be shown giving the parameter estimate, standard error, t test, and p-value. The results will be given for each analysis.

 3.1 Original Respondent Sample

Categorical Variables

Continuous Variables

Logistic Regression Model

3.2 Respondent Sample with Substitutes (Final Sample)

Categorical Variables

Continuous Variables

Logistic Regression Model

3.3 Final Sample with Nonresponse Adjustments Applied

Categorical Variables

Continuous Variables

4.        CONCLUSIONS

A summary of the results will be presented along with a conclusion on the effect of substitutes and the non-response weighting adjustment.

References

Research Triangle Institute (2012). SUDAAN Language Manual, Volumes 1 and 2, Release 11.

Research Triangle Park, NC: Research Triangle