TIMSS 2019 Main Study Recruitment & Field Test Update - Changes Memo

DATE: December 22, 2017

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, NCES

FROM: Lydia Malley, NCES

Re: Trends in International Mathematics and Science Study (TIMSS 2019) Main Study Recruitment and Field Test Change Request (OMB# 1850-0695 v.11)

The Trends in International Mathematics and Science Study (TIMSS) is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years, with the next TIMSS assessment, TIMSS 2019, being the seventh iteration of the study. TIMSS provides a comparison of U.S. student performance with those of their international peers in mathematics and science at grades 4 and 8. TIMSS is coordinated by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that creates the assessment framework, assessments, and background questionnaires and provides procedures and technical standards which all countries must follow. In the U.S., the National Center for Education Statistics (NCES) conducts TIMSS. In preparation for the TIMSS 2019 main study, NCES will conduct a field test in 2018 to evaluate new assessment items and background questions, to ensure practices that promote low exclusion rates, and to ensure that classroom and student sampling procedures proposed for the main study are successful. The U.S. TIMSS 2019 field test recruitment began in May 2017, with data collection conducted from March through April 2018. The U.S. TIMSS 2019 main study recruitment will begin in May 2018, with data collection conducted from March through May 2019. The request for the TIMSS 2019 Main Study recruitment & Field Test was approved in July 2017 (OMB# 1850-0695 v.10).

This request is to provide additional recruitment materials that will be used during the later portion of field test recruitment and throughout the main study, and to provide the final international versions of the TIMSS 2019 field test questionnaires including the proposed U.S. adaptations. The IEA has released the final international versions of the TIMSS 2019 field test questionnaires, and is in the process of approving the adaptations to be made to the U.S. versions of the field test questionnaires. If any changes are made to the U.S. adaptations provided in this submission, the final versions of the adapted U.S. versions of the TIMSS 2019 field test questionnaires will be provided to OMB as a change request in January 2018.

These revisions described in this submission do not affect the estimated burden to respondents or the total cost to the federal government for TIMSS 2019 (as approved in OMB# 1850-0695 v.10).

Supporting Statement Part A:

Changes in Part A include updating the information in the Preface to refer to the latest changes in this submission and the deletion of the potential for NCES to conduct cognitive interview on some of the items. The cognitive interviews are no longer necessary because the items that were being considered for cognitive testing are being tested as part of the field test. Finally, under the student questionnaire section, the mode has changed from online to paper-and pencil for the field test only. This is due to the time constraints and the international contractor needing to focus on the technical aspect of the student assessment for the field test.

Page 2 (Preface):

“The National Center for Education Statistics (NCES), within the U.S. Department of Education (ED), is requesting clearance for data collection materials and procedures for the 2019 Trends in International Mathematics and Science Study (TIMSS 2019) main study recruitment and field test, both due to begin in spring 2018. The Supporting Statements Part A and B in this submission are based on those approved in December 2016 and revised in April 2017 for TIMSS 2019 Field Test Recruitment and Pilot Test(OMB# 1850-0695 v.8-9), with the following changes:

• Total requested burden estimates in Part A were expanded to include burden for the field test data collection and main study recruitment; and

• Sampling information for the TIMSS main study was added to Part B.

Appendix A provides TIMSS 2019 state, district, and school communication materials approved for the field test (OMB# 1850-0695 v.8-9) and newly added for the main study (adapted from the field test materials). Parental notification materials and consent forms are provid­ed in Appendix B, and draft versions of the TIMSS 2019 field test questionnaires (TIMSS 2015 questionnaires) are provid­ed in Appendix C.”

was replaced with:

“The National Center for Education Statistics (NCES), within the U.S. Department of Education (ED), received approval in July 2017 for data collection materials and procedures for the 2019 Trends in International Mathematics and Science Study (TIMSS 2019) main study recruitment and field test, both due to begin in spring 2018 (OMB# 1850-0695 v.10). This request is to provide additional recruitment materials that will be used during the later portion of field test recruitment and throughout the main study, and to provide the final international versions of the TIMSS 2019 field test questionnaires including the proposed U.S. adaptations.

Appendix A-B provides TIMSS 2019 state, district, and school communication materials and parent notification materials approved for the field test and for the main study (OMB # 1850-0695 v.10). In this submission, Appendix A-B includes newly added communication materials for the later portion of field test recruitment and for the main study. Draft U.S. versions of the TIMSS 2019 field test questionnaires are provid­ed in Appendix C, with a summary of changes to the instruments from TIMSS 2015 Main Study to the TIMSS 2019 Field Test. The International Association for the Evaluation of Education Achievement (IEA) has released the final international versions of the TIMSS 2019 field test questionnaires, and is in the process of approving the adaptations to be made to the U.S. versions of the field test questionnaires. If any changes are made to the U.S. adaptations provided in this submission, the final versions of the adapted U.S. versions of the TIMSS 2019 field test questionnaires will be provided to OMB as a change request in January 2018.”

Page 3 (Preface):

“cognitive labs at grades 4 and 8, and” was taken out of: “In addition to the field test, NCES may conduct cognitive labs at grades 4 and 8, and focus groups of school principals to examine ways to increase participation.”

Page 5 (A.2. Purposes and Uses of Data – TIMSS 2019 Components – Questionnaires):

“Draft international versions of the TIMSS 2019 field test questionnaires will become available in December 2017, at which point NCES will propose U.S. adaptations to those questionnaires, which are expected to be finalized in January 2018. TIMSS 2015 final questionnaires are provided in Appendix C.”

was replaced with:

“Draft international versions of the TIMSS 2019 field test questionnaires became available in December 2017, and NCES has proposed to IEA the U.S. adaptations to those questionnaires, which are expected to be finalized in January 2018. The TIMSS 2019 final international versions of the field test questionnaires with the proposed U.S. adaptations are provided in Appendix C.”

Page 5 (A.2. Purposes and Uses of Data – TIMSS 2019 Components – Student Questionnaires):

“During the field test, student questionnaire will be administered online only, while during the main study a subset of students will be administered a paper-and-pencil version of (1) the student questionnaire and (2) the same test booklets used in the 2017 pilot equivalence study (booklets that consist of the TIMSS trend items carried over from TIMSS 2015).”

was replaced with:

“During the field test, student questionnaire will be administered on paper due to the need of the international contractor to focus on the technical aspect of the assessment for the field test. During the main study the questionnaire will be administered online with a subset of students to be administered a paper-and-pencil version of (1) the student questionnaire and (2) the same test booklets used in the 2017 pilot equivalence study (booklets that consist of the TIMSS trend items carried over from TIMSS 2015).”

Appendix A-B:

Changes in Appendix A-B consist of the contact information having been updated throughout and of the addition of the following materials to both the field test and main study sections:

Appendix A1 and A2

Added Communication Material	Appendix A1 Page #	Appendix A2 Page #
Email to School Coordinator (SC) for MyTIMSS Registration and to Provide School Information (PSI)	16	89
MyTIMSS Registration and Provide School Information (PSI) Guide	18	91
Automated Email for all MyTIMSS registrants	22	95
Automated Email for MyTIMSS School Coordinator registrant	23	96
MyTIMSS Login Page	24	97
MyTIMSS Home Page	26	99
MyTIMSS Provide School Information (PSI) Page	27	100
Submit Class List Page – Grade 4	31	108
Submit Student List Page – Grade 4	36	109
Submit Class List Page – Grade 8	40	117
Submit Student List Page – Grade 8	45	118
MyTIMSS Prepare for Assessment Page	46	119
Student-Teacher Linkage Form (STLF) Email	47	120
Student Tracking Form (STF) and Teacher Tracking Form (TTF) Email	48	121
Instructions for Student Tracking Form (STF) and Teacher Tracking Form (TTF)	49	122
Student-Teacher Linkage Form (STLF) Template	51	124
Student Tracking Form (STF) Template	52	125
Teacher Tracking Form (TTF) Template	53	126
MyTIMSS Documents Page	54	127
MyTIMSS Important Websites and Contact Us Pages	55	128
School Principal Questionnaire Login Card/Email	56	129
Teacher Questionnaire Login Card/Email	57	130
Online Questionnaire Login Screen Text	58	131
Student Appointment Card for TIMSS 2018 FT: Grade 8 only	59	132
Student Login Card	61	134
Student Certificate of Completion	62	135
School Certificate of Completion	63	136

Appendix B1 and B2

Added Communication Material	Appendix B1 Page #	Appendix B2 Page #
Multi-translation Notice for Parents	75	148

These additional materials complete all of the communication materials that will be used throughout the field test and for main study recruitment.

Appendix C:

Appendix C has been updated from the TIMSS 2015 main study versions of the questionnaires to include the TIMSS 2019 field test U.S. versions of the questionnaires. The IEA has released the final international versions of the TIMSS 2019 field test questionnaires, and is in the process of approving the adaptations to be made to the U.S. versions of the field test questionnaires. In this submission, Appendix C has been broken into two parts: Appendix C.1 lists the changes that were made to the TIMSS 2019 U.S. versions of the field test questionnaire from the last round of TIMSS (TIMSS 2015), and Appendix C.2 provides the draft U.S. adapted versions of the TIMSS 2019 field test questionnaires to be administered in spring 2018. Appendix C.1 is based on a comparison of the draft TIMSS 2019 U.S. field test questionnaires with the final U.S. versions of the TIMSS 2015 main study questionnaires, which were included in the last TIMSS 2019 Main Study Recruitment and Field Test submission (OMB# 1850-0695 v.10) because the TIMSS 2019 field test international questionnaires were not yet available. If any changes are made to the U.S. adaptations provided in this submission, the final versions of the adapted U.S. versions of the TIMSS 2019 field test questionnaires will be provided to OMB as a change request in January 2018.

The changes detailed in Appendix C.1 are listed first by grade, then by questionnaire (e.g., school principal, teacher, and student), and lastly by the four types of changes made: new U.S. adaptations (not already included in the 2015 main study); new TIMSS items; deleted TIMSS items; and revisions to TIMSS item wording (made for the TIMSS 2019 field test administration).