Part A TIMSS 2019 Main Study Recruitment & Field Test

The National Center for Education Statistics (NCES), within the U.S. Department of Education (ED), is requesting clearance for data collection materials and procedures for the 2019 Trends in International Mathematics and Science Study (TIMSS 2019) main study recruitment and field test, both due to begin in spring 2018. The Supporting Statements Part A and B in this submission are based on those approved in December 2016 and revised in April 2017 for TIMSS 2019 Field Test Recruitment and Pilot Test(OMB# 1850-0695 v.8-9), with the following changes:

Total requested burden estimates in Part A were expanded to include burden for the field test data collection and main study recruitment; and
Sampling information for the TIMSS main study was added to Part B.

Appendix A provides TIMSS 2019 state, district, and school communication materials approved for the field test (OMB# 1850-0695 v.8-9) and newly added for the main study (adapted from the field test materials). Parental notification materials and consent forms are provided in Appendix B, and draft versions of the TIMSS 2019 field test questionnaires (TIMSS 2015 questionnaires) are provided in Appendix C.

TIMSS is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years (1995, 1999, 2003, 2007, 2011, and 2015), with the next TIMSS assessment, TIMSS 2019, being the seventh iteration of the study. In TIMSS 2015, 48 countries and 6 other education systems participated at grade 4, and 38 countries and 6 other education systems participated at grade 8. The United States will participate in TIMSS 2019 to continue to monitor the progress of its students compared to that of other nations and to provide data on factors that may influence student achievement.

New in 2019, TIMSS will be a technology-based assessment conducted in an electronic format (referred to as “eTIMSS”) and administered using the eTIMSS Assessment Platform. There are two primary goals for the transition to a technology-based assessment: (1) maintaining continuity to measure trends with the past paper-and-pencil TIMSS assessments, and (2) developing relevant assessment innovations that take advantage of new technologies.

TIMSS is led by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the frameworks used to develop the assessment, the survey instruments, and the study timeline. IEA decides and agrees upon a common set of standards, procedures, and timelines for collecting and reporting data, all of which must be followed by all participating countries. As a result, TIMSS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., NCES conducts this study in collaboration with the IEA and a number of contractors (Westat, Avar Consulting (Avar), AIR, and Hager Sharp) to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in TIMSS is consistent with NCES’s mandate of acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543)].

Compared to previous assessment cycles, TIMSS 2019 differs in several ways:

Unlike TIMSS 2015, the IEA will not conduct TIMSS Advanced.
TIMSS 2019 will be a digitally-based assessment administered on the eTIMSS Assessment Platform.
TIMSS 2019 administered a pilot study in 2017, which will be followed by a field test in 2018.

The United States participated in the 2017 pilot study to assist in the development of eTIMSS. The pilot is a very important part of the transition from a paper-based assessment to a technology-based assessment. The pilot was designed to provide information that can be used to reduce the data collection burden in 2019. The purpose of the pilot study was to (1) try out the new eTIMSS Assessment Platform, using the tablet player, data monitoring system, and online scoring; (2) conduct a mode effect study to examine the effect of administering the assessment on tablet versus paper; and (3) pilot newly-developed problem-solving and inquiry assessment items. Data collection for the pilot occurred in April and early May 2017. Student samples were obtained by selecting two classes from each school (e.g., two mathematics classes for grade 8).

In preparation for the TIMSS 2019 main study, all countries are asked to implement a 2018 field test. The purpose of the TIMSS field test is to evaluate new assessment items and background questions to ensure practices that promote low exclusion rates, and to ensure that classroom sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively. Additionally, TIMSS staff will work to help respondents understand the study’s value relative to burden imposed, and to ensure a high level of school participation in the main study.

Because TIMSS is a collaborative effort among many parties, the United States must adhere to the international schedule set forth by the IEA, including the availability of final field test and main study plans as well as draft and final questionnaires. In order to meet the international data collection schedule, to align with recruitment for other NCES studies (e.g., the National Assessment of Education Progress, NAEP), and for schools to put the TIMSS 2019 assessment on their calendars, recruitment activities for the field test must begin in May 2017, and recruitment for the main study in May of 2018. Data collection for the field test will occur from March through April 2018. The U.S. TIMSS main study will be conducted from April through May, 2019. Due to the overlap in the timing of activities, in addition to requesting to conduct the field test data collection and main study recruitment, this submission carries over the approved request for the 2018 field test recruitment (OMB# 1850-0695 v.8-9).

In December 2017, NCES will submit a request with a 30-day public comment period notice published in the Federal Register, which will provide the final international versions of the field test instruments for data collection in March-April 2018, along with draft U.S. adaptations for these instruments. The final versions of the TIMSS 2019 field test instruments, including finalized U.S. adaptations, will be submitted to OMB as a change request in January 2018. In late 2018, NCES will submit another request, with a 30-day public comment period notice published in the Federal Register, which will include the final international versions of the main study instruments for data collection in April-May 2019, along with draft U.S. adaptations for these instruments. The final versions of the TIMSS 2019 main study instruments, including finalized U.S. adaptations, will be submitted to OMB as a change request in January 2019. The main study data collection instruments will be a subset of the field test instruments.

In addition to the field test, NCES may conduct cognitive labs at grades 4 and 8, and focus groups of school principals to examine ways to increase participation. In such case, a separate OMB request will be submitted for these activities under NCES’s generic clearance for developmental studies (OMB# 1850-0803).

A. Justification

A.1 Importance of Information

Benchmarking of U.S. student achievement against other countries continues to be of high interest to education policymakers, and informs policy discussions of economic competitiveness and workforce and post-secondary preparedness. TIMSS provides a unique opportunity to compare U.S. students‘ mathematics and science knowledge and skills at fourth and eighth grade with that of their peers in countries around the world. Science, technology, engineering, and mathematics (STEM) preparedness is key to economic improvement.

The continuation of U.S. participation allows for the study of past and current education policies that have shaped science and mathematics achievement over the past 24 years. Furthermore, participating countries are not only able to obtain information about students' knowledge and abilities in the specified subjects, but also about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement in the respective subject areas.

TIMSS complements what we learn from national assessments such as the National Assessment of Educational Progress (NAEP) by identifying the strengths and weaknesses of student science and mathematics achievement relative to participating countries around the world. It provides valuable benchmarking information about educational polices enacted in other countries and policies that could be applied to U.S. educational practices.

Based on earlier TIMSS data releases, it is likely that the results of TIMSS 2019 will draw great attention in the United States and elsewhere. It is therefore expected that TIMSS will contribute to ongoing national and international debates and efforts to improve mathematics and science learning and achievement.

A.2 Purposes and Uses of Data

TIMSS assesses mathematics and science knowledge and skills at grades 4 and 8. TIMSS is designed to align broadly with curricula in the participating countries. The results, therefore, suggest the degree to which students have learned concepts and skills likely to have been taught in school. TIMSS also collects background information on students, teachers, schools, curricula, and official education policies in order to allow cross-national comparison of educational contexts that may be related to student achievement.

Data compiled and collected from TIMSS 2019 allows for evidence-based decisions to be made for the purposes of educational improvement. Each successive participation in TIMSS provides trend information about student achievement in mathematics and science relative to other countries, as well as indicators that show how this achievement relates to demographic and curricular, school, teacher, and student factors that provide the educational context for achievement. This high quality, internationally comparative trend data provide key information to inform education policy discussions.

Through participation in TIMSS and other international assessment programs, NCES is able to provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance, and to suggest hypotheses about the relationship between student performance and factors that may influence performance as well as areas in which students have strengths or weaknesses. The international studies identify differences among countries over time in instructional practices, school policies, and opportunity-to-learn that informs discussions about how to organize instruction.

NCES’s mandate [Section 406 of the General Education Provisions Act, as amended (20 U.S.C. §1221e-1)] specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the United States and in other nations." and the Educational Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543) specifies that NCES shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations. TIMSS is essential for any international perspective on students’ mathematics and science knowledge and skills, and U.S. participation in ICILS is aligned with both the national and international aspects of NCES' mission

TIMSS 2019 Components

The mathematics and science assessments at grades 4 and 8 are organized around a content dimension that specifies the subject matter to be assessed and a cognitive dimension that specifies the thinking processes to be assessed. The cognitive domains are the same in mathematics and science: knowing, applying, and reasoning. The TIMSS 2019 frameworks will be published in August 2017 and will be similar to the TIMSS 2015 frameworks, which will be slightly revised to provide more specificity for item writers, and to better reflect current curricula in participating countries. It is not anticipated that there will be any revisions to the content domains or cognitive domains, nor to the target percentages for the content domains or cognitive domains at either subject at either grade.

In fourth-grade mathematics the cognitive content domains include: number, geometric shapes and measures, and data display. More advanced content in these three domains are assessed in eighth grade, supplemented by a data and chance domain. TIMSS assesses a range of problem-solving situations within mathematics, with about two-thirds of the questions requiring students to use applying and reasoning skills.

In science at fourth grade, the content domains include: life science, physical science, and earth science. At eighth grade the content domains transition to a more discipline-based approach, reflecting the differences in instruction from elementary school. The content domains at eighth grade are: biology, chemistry, physics, and earth science. TIMSS 2019 will also measure science practices and science inquiry, reflecting recent emphasis on these skills in many countries’ curricula and content standards.

Assessment Mode

In TIMSS 2019 countries can choose to administer a digitally-based assessment, rather than the traditional paper-and-pencil mode of assessment. The United States will participate in digitally-based TIMSS. Studies will be conducted in the 2017, 2018, and 2019 to operationalize the new digital mode of assessment, as well as to analyze mode effects so that the important TIMSS trend reporting can be maintained. The assessments will be conducted on tablets, such as the Microsoft Surface Pro, with a stylus and external keyboard.

Assessment Instruments

In order to minimize burden and to ensure broad subject-matter coverage, TIMSS will use a matrix sampling approach where the mathematics and science items at each grade level are organized into a set of test booklets, with each student taking only one booklet. Test items are either multiple-choice or constructed response items.

Questionnaires

The background questionnaires for TIMSS 2019 are being developed to address a background questions framework developed internationally. The United States will adapt the questions to fit the U.S. education context, including adding a few questions, such as about the race/ethnicity of students. Teacher and school questionnaires will be offered online, with a paper-and-pencil backup. All students in the field test, and most students in the main study, will answer their questionnaires on the tablets, along with the cognitive assessment questions. In the main study, a subset of 1,500 students per grade level will take a paper TIMSS assessment (for the purposes of a bridging study between the paper and new electronic version). TIMSS 2019 field test questionnaires are expected to be largely based on TIMSS 2015 questionnaires. Draft international versions of the TIMSS 2019 field test questionnaires will become available in December 2017, at which point NCES will propose U.S. adaptations to those questionnaires, which are expected to be finalized in January 2018. TIMSS 2015 final questionnaires are provided in Appendix C.

School Questionnaire. A representative from each participating school will be asked to provide information on mathematics and science resources, teacher availability and retention, principal leadership, school emphasis on academic success, school climate, and parental involvement in school activities. The TIMSS school questionnaire is expected to take 20 minutes to complete and will be offered online, with a paper-and-pencil backup. The school questionnaire was not administered during the pilot study.

Teacher Questionnaire. At grades 4 and 8, mathematics and science teachers of students in selected classes will be asked to complete a teacher questionnaire. Teacher questionnaires will include questions about teacher preparation and experience, mathematics and science topics taught, instructional resources and technology, instructional time, instructional engagement, classroom assessment, and technology resources and instruction in their classes. The teacher questionnaire is expected to take 30 minutes to complete and will be offered online, with a paper-and-pencil backup. The teacher questionnaire was not administered during the pilot study.

Student Questionnaire. Student information will be collected about home resources, student motivation, self-concept, self-efficacy, and student characteristics such as gender and race/ethnicity. The student questionnaire is expected to take 30 minutes to complete and is administered after the cognitive assessment on the same digital device. During the pilot study we administered only an abbreviated student questionnaire that took about 10 minutes to complete and collected limited demographic and background information. During the field test, student questionnaire will be administered online only, while during the main study a subset of students will be administered a paper-and-pencil version of (1) the student questionnaire and (2) the same test booklets used in the 2017 pilot equivalence study (booklets that consist of the TIMSS trend items carried over from TIMSS 2015).

A.3 Improved Information Technology (Reduction of Burden)

The TIMSS 2019 design and data collection procedures are prescribed internationally including student assessments and questionnaires being administered on tablets. TIMSS also administered paper-and-pencil versions of the assessment in the 2017 pilot test and will administer paper versions to a subsample of students in the 2019 main study to study the mode effect of transitioning from paper to digital administration. Each participating nation is expected to adhere to the internationally prescribed design. In the United States, the school and teacher questionnaires will be made available to school administrators and teachers online as the main mode of administration, with a paper-and-pencil backup to facilitate user preference for participation.

A communication website, MyTIMSS USA, was used during the 2017 pilot, and will be used for the 2018 field test and the 2019 main study in order to provide a simple, single source of information to engage sample schools and maintain high levels of their involvement. This portal will be used throughout the assessment cycle to inform schools of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather class and student lists from participating schools electronically using an adaptation of Westat’s secure E-filing process. E-filing is an electronic system for submitting lists of student information, including student background information in school records. Instructions to school coordinators on how to submit class and student lists are included in Appendix A. E-filing has been used successfully in NAEP for more than 10 years, and was used in TIMSS 2015 and the PISA 2012 and 2015 assessments. The E-filing system provides advantageous features such as efficiency and data quality checks. Schools will access the E-filing system through the MyTIMSS web site.

The eTIMSS assessments will be implemented using tablets carried into schools by the data collection staff so that TIMSS will not need to use school equipment. Schools can continue to use their own equipment for instruction and assessment, and TIMSS will not need to burden school or district IT staff to set up school equipment or take down firewalls. TIMSS does not require internet access during the assessment. Student data are uploaded to Westat secure servers after the assessment.

A.4 Efforts to Identify Duplication

In the United States, mathematics and science achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971 through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. TIMSS 2019 does not duplicate these assessments.

TIMSS 2019 is part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent the U.S. participation in international studies involving approximately 60-65 countries. As part of international cooperative studies, the United States must collect the same information at the same time as the other participating nations for purposes of making both valid international comparisons with other countries and with the previous TIMSS data. While some studies in the United States collect similar, though not identical, types of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in TIMSS in that they do not allow for comparisons outside the United States. Furthermore, the data collected through TIMSS is based on unique frameworks that are not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the United States must agree to administer the same core instruments that are administered in all other participating countries. Because the items measuring mathematics and science achievement have been developed with intensive international coordination, any changes to the instruments require international coordination and approval.

A.5 Minimizing Burden for Small Entities

No small entities are part of this sample. The school samples for TIMSS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. All school sizes are needed to ensure an appropriate representation of each type of school in the selected sample of schools. Burden will be minimized wherever possible. For example, schools will be selected so as to avoid overlap with other NCES assessments such as NAEP and PISA. In addition, contractor staff will conduct all test administrations, provide all equipment, and will assist with parental notification, sampling, and other tasks as much as possible within each school.

A.6 Frequency of Data Collection

The pilot data collection occurred in April and early May 2017. The field test data collection is scheduled for March 5 through April 15, 2018. The full-scale data collection is scheduled for April through May 2019. This schedule is prescribed by the international collective for TIMSS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries as well as to maintain trend lines.

A.7 Special Circumstances

None of the special circumstances identified in the Instructions for Supporting Statement apply to the TIMSS study.

A.8 Consultations outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An International Steering Committee has general oversight of the study and each National Research Coordinator participates in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened separate panels of mathematics and science experts from around the world to develop cognitive items.

The majority of the consultations (outside NCES) have involved the IEA-Amsterdam, in the Netherlands, the IEA-Data Processing Center (DPC) in Hamburg, Germany, and the TIMSS International Study Center (ISC) at Boston College in the United States. Key to these ongoing consultations are: Dirk Hastedt (executive director of the IEA); Oliver Neuschmidt (head of the IEA Data Processing and Research Center); Michael Martin, Ina V.S. Mullis, Victoria Centurino, and Kerry Cotter (directors of the ISC TIMSS teams), all of whom have extensive experience in developing and operating international education surveys (especially related to TIMSS).

A.9 Payments or Gifts to Respondents

In order to achieve acceptable school response rates in international studies, schools in the U.S. are usually offered $200 to thank them for their participation and the time they invest in and the space they make available for the international assessments. High response rates are required by both IEA and NCES, and are difficult to achieve in school-based studies. The U.S. has historically had difficulties in achieving sufficient participation levels. Based on incentives provided in past administrations of TIMSS and currently offered in other international assessments, schools will be offered $200 for their time in the field test and the main study. In the 2017 eTIMSS pilot, students took both the paper-and-pencil version and an eTIMSS version of the assessment. Student time was twice what it typically is, and involved more assistance from the school staff in order to administer either two sessions in one day, or over two days. Therefore, schools participating in the eTIMSS pilot were offered $400 for their time, help, and participation.

The school staff serving as School Coordinators will receive $100 for their time and effort. The School Coordinator serves a critical role in data collection, functioning as the central school contact and facilitating arrangements for the assessments. They are asked to file class and student listing forms; arrange the date, time and space for the assessment; and disseminate information to parents and students.

For the first time, TIMSS will offer $20 to teachers who complete the teacher questionnaires. Both PISA and PIRLS (in 2016) have offered this level of incentive to teachers who complete a 30-minute questionnaire. In the past, field staff and Westat home office staff have spent considerable time tracking down teachers after the assessment to obtain a reasonable response rate. The teacher questionnaire was not administered during the pilot test and thus no teacher incentive was used.

Consistent with prior administrations of TIMSS, as a token of appreciation for their participation, students will receive a small gift valued at approximately $4. In TIMSS 2015, each participating student received a small flashlight that could be clipped with an attached karabiner to a backpack or belt loop. Students will also receive a certificate with their name thanking them for participating and representing the United States in TIMSS. In schools where it is permitted, eighth-grade students participating in TIMSS may also receive a certificate from the U.S. Department of Education for four hours of volunteer service. Additionally, some schools also offer recognition parties with pizza or other treats for students who participate; however these are not reimbursed by NCES or the contractor.

A.10 Assurance of Confidentiality

Data security and confidentiality protection procedures have been put in place for TIMSS to ensure that the TIMSS contractor for the U.S. and its subcontractors comply with all privacy requirements, including:

The statement of work of this contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Family Educational and Privacy Act (FERPA) of 1974 (20 U.S.C. §1232(g));
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Confidential Information Protect and Statistical Efficiency Act of 2002;
E-Government Act of 2002, Title V, Subtitle A;
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement for this study.

Furthermore, the contractor will comply with the Department’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to the revised NCES Statistical Standards, as described at the website: http://nces.ed.gov/statprog/2012/.

The laws pertaining to the use of personally identifiable information are clearly communicated in correspondence with states, districts, schools, teachers, students, and parents. Letters and information materials will be sent to parents and school administrators describing the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential (see copies in appendix A-B).

Letters to teachers, school coordinators, and supporting materials will read (Appendix A-1 and A-2):

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Letters to states, districts, and schools and parent notification letters and supporting materials will read (Appendix A-1, A-2, B-1, and B-2):

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided by school staff and students may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

The following statement will appear on the login page for eTIMSS, MyTIMSS, and the front cover of the printed questionnaires (the phrase “search existing data resources, gather the data needed” will not be included on the student questionnaire):

The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United States as authorized by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0695. The time required to complete this information collection is estimated to average [XX] minutes per [respondent type], including the time to review instructions [, search existing data resources, gather the data needed,] and complete and review the information collection. If you have any comments or concerns regarding the accuracy of the time estimate(s), suggestions for improving the form, or questions about the status of your individual submission of this form, write directly to: Trends in Mathematics and Science Study (TIMSS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, 4^th floor, Washington, DC 20202.

OMB No. 1850-0695, Approval Expires xx/xx/2020.

The TIMSS 2019 confidentiality plan includes signing confidentiality agreements and notarized nondisclosure affidavits by all contractor and subcontractor personnel and field workers who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility. Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included on any data file. In eTIMSS, students log into the automated assessment using non-identifying ID’s and thus the resulting data are collected and stored with only the non-identifying TIMSS assigned ID. The data are collected from the tablets using a process that encrypts the data, and uploads the data onto a Westat secure FTP site.

NCES understands the legal and ethical need to protect the privacy of the TIMSS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure analysis of the TIMSS 2019 data when preparing the data files for use by researchers, in compliance with ESRA 2002 (20 U.S.C. §9573). Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including swapping data and omitting key identification variables (i.e., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or ZIP Code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files.

A.11 Sensitive Questions

The questionnaires do not have items considered to be of a sensitive nature.

A.12 Estimates of Burden

This request is to conduct TIMSS 2019 main study recruitment and field test. Due to the overlap in the timing of activities, this submission also carries over the approved request and burden for the 2018 field test recruitment (OMB# 1850-0695 v.8-9). Therefore, the burden estimates include burden for (1) contacting states, districts, schools, and parents in order to recruit for the TIMSS 2019 field test and main study, including (a) sending recruitment letters to districts and schools selected for each study, (b) contacting and seeking research approvals from special handling districts, where applicable, and (c) notifying parents of sampled students about their participation in either study; and (2) collecting field test data. The estimated burden for these efforts is shown in Table A.1.

The district and school contact letters for the field test and main study are assumed to impose small burden on all contacted parties, both those that refuse and those that agree to participate in the TIMSS studies. Thus the burden hours for this activity are based on the total number of districts and schools contacted rather than the total number agreeing to participate.

The special handling districts are those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous TIMSS data collections, we estimate that there may be up to 10 special handling districts in the sample. Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. This operation should begin in the spring of the year preceding the start of the data collection to allow sufficient time for special handling districts’ review processes. We will continue to work with these districts until we receive a final response from each district (approval or denial of request) up until March 1, 2018 for the field test and March 1, 2019 for the main study.

The total district and school response burden estimate for the main study recruitment and field test is based on 10 minutes for districts to read materials and respond, and 20 minutes for schools to read materials and respond. The total response burden estimate for IRB approvals is based on 120 minutes for staff approval and 60 minutes for panel approval. The total response burden for parental notification is based on 10 minutes for reading and reviewing recruitment materials and notification forms. The total response burden for the field test data collection is based on 30 minutes for students and teachers to complete a questionnaire, and 20 minutes for a school administrator to complete a school questionnaire.

Please note that the questionnaire burden for students in Table A.1. is estimated at 50 minutes because experience has shown that administering the 30-minute student questionnaire after the assessment often takes longer due to time for bathroom breaks after the assessment and getting the class of students resettled to focus on the questionnaire. Based on the estimated hourly rates for principals/administrators, school coordinators, teachers, and parents of $45.86, $23.38, $23.38, and $23.86, respectively¹, and the federal minimum wage of $7.25 as the hourly rate for the students, and based on the estimated total of 10.974 burden hours for TIMSS 2019 main study recruitment and field test recruitment and data collection, the estimated respondent burden time cost is $217,348.

Table A.1. Burden estimates for TIMSS 2019 field test data collection in 2018 and for TIMSS 2019 field test and main study recruitment, in 2017 and 2018, respectively, for grades 4 and 8.

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Per respondent (minutes)	Total burden (hours)
Field Test Recruitment
Contacting Districts	80	1.00	80	80	10	14
Contacting Schools	80	1.00	80	80	20	27
District IRB Staff Study Approval	10	1.00	10	10	120	20
District IRB Panel Study Approval	60	1.00	60	60	60	60
Parental notification	2,800	1.00	2,800	2,800	10	467
Field Test Recruitment Burden			3,030	3,030		588
Field Test Data Collection
Student
Grade 4
Directions	1,550	0.93	1,442	1,442	20	481
Assessment	1,550	0.93	1,442	1,442	72	1,731
Student Questionnaire	1,550	0.93	1,442	1,442	50	1,202
Grade 8
Directions	1,550	0.93	1,442	1,442	20	481
Assessment	1,550	0.93	1,442	1,442	90	2,163
Student Questionnaire	1,550	0.93	1,442	1,442	50	1,202
Total Student Burden Field Test			2,884	5,768		3,366
School Staff (Grades 4 and 8)
School Administrator	80	0.95	76	76	30	38
Teacher	320	0.95	304	304	30	152
School Coordinator (Grade 4 & 8)	80	1.00	80	80	240	320
Total School Burden Field Test			460	460		510
Field Test Data Collection Burden			6,374	9,258		4,464
Main Study Recruitment
Contacting Districts	670	1.00	670	670	10	112
Contacting Schools	670	1.00	670	670	20	224
District IRB Staff Study Approval	84	1.00	84	84	120	168
District IRB Panel Study Approval	504	1.00	504	504	60	504
Parental notification	29,480	1.00	29,480	29,480	10	4,914
Main Study Recruitment Burden			31,408	31,408		5,922
Main Study Data Collection
Student
Grade 4
Directions	14,740	0.93	13,709	13,709	20	4,570
Assessment	14,740	0.93	13,709	13,709	72	16,451
Student Questionnaire	14,740	0.93	13,709	13,709	50	11,425
Grade 8
Directions	14,740	0.93	13,709	13,709	20	4,570
Assessment	14,740	0.93	13,709	13,709	90	20,564
Student Questionnaire	14,740	0.93	13,709	13,709	50	11,425
Total Student Burden Main Study			27,418	54,836		31,990
School Staff (Grades 4 and 8)
School Administrator	670	0.95	637	637	30	319
Teacher	2,680	0.95	2,546	2,546	30	1,273
School Coordinator (Grade 4 & 8)	670	1.00	670	670	240	2,680
Total School Burden Main Study			3,853	3,853		4,272
Main Study Data Collection Burden			62,679	90,097		42,184
Total burden requested in this submission			40,812	43,696		10,974

Note: Total burden requested in this submission includes burden associated with field test data collection and with recruiting schools and contacting districts and parents for the TIMSS 2019 field test and main study (items in black font). Cells showing burden that was approved by OMB for the 2018 field test recruitment (OMB# 1850-0695 v.8-9) are shaded in light gray and included in the requested burden totals. Burden that will be requested for the future TIMSS 2019 main study data collection or that is estimated for the field test cognitive assessments, which are not subject to PRA, are demarcated in gray font, are presented here for information only purposes, and are not included in burden totals.

A.13 Total Annual Cost Burden

There are no additional costs to respond beyond the time to respond.

A.14 Annualized Cost to Federal Government

The cost to the federal government for conducting initial phases of TIMSS 2019, including 2018 field test operations (preparations, data collection, and scoring) and 2019 main study recruitment is estimated to be $1,930,283 over a 2-year period (see table breakdown below). These figures include all direct and indirect costs.

Components with breakdown	Estimated costs
PILOT (2017)
Recruitment	95,000
Preparations (e.g., adapting instruments, sampling)	177,851
Data collection, scoring, and coding	323,822
FIELD TEST (2018)
Recruitment	149,885
Preparations (e.g., adapting instruments, sampling)	409,276
Data collection, scoring, and coding	949,635
MAIN STUDY (2019)
Recruitment	571,372
Preparations (e.g., adapting instruments, sampling)	53,278
Data collection, scoring, and coding	4,961,896
Current package components	$1,930,283
Grand total	$7,692,015

Note: Cells showing the cost to federal government for the TIMSS activities that were approved by OMB in prior submissions (OMB# 1850-0695 v.8-9) are shaded in light gray and are not included in the “Current package components” total. The cost for the activities that will be requested for the future TIMSS 2019 main study data collection are demarcated in gray font.

A.15 Program Changes or Adjustments

The apparent increase in burden from the last approval is due to the fact that the last request was to conduct the TIMSS 2019 pilot test and to recruit schools and communicate with districts and parents in preparation for the 2018 TIMSS 2019 field test, while this request is to conduct the TIMSS 2019 field test and to recruit schools and communicate with districts and parents in preparation for the TIMSS 2019 main study.

With respect to the previous administration of Main Study TIMSS, TIMSS 2019 differs from TIMSS 2015 in the following ways:

No federally funded state benchmarking;
Transition to a digitally-based assessment in 2019;
Addition of a pilot study preceding the field test; and
TIMSS Advanced will not be administered in 2019.

A.16 Plans for Tabulation and Publication

The TIMSS pilot and field test are designed to provide a statistical review of the performance of items on the cognitive assessment and questionnaires in preparation for the main study data collection. They will also provide valuable experience in administering eTIMSS on tablets and the performance of TIMSS items in a digitally-based environment. Corresponding paper-and-pencil assessments in 2017 and 2019 will provide information for evaluating the mode effect of the transition from paper to eTIMSS.

Based on the data collected in the main study, the TIMSS International Study Center will prepare separate reports for mathematics and science at grades 4 and 8. These reports will be released by December 2020. As has been customary, NCES will also release a report for each study at the same time as the international reports are released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of either dataset. Examples of past reports on TIMSS can be found at http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2013009. In the spring of 2021, the International Study Center will also prepare technical reports for TIMSS 2019, describing the design and development of the assessments as well as the scaling procedures, weighting procedures, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for each study.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of TIMSS 2019 results in the United States are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

Dates	Activity
April—December 2016	Prepare data collection manuals, forms, assessment materials, and questionnaires for pilot
July—August 2016	Select school samples for pilot
October 2016—January 2017	Contact and gain cooperation of states, districts, and schools for pilot
February—March 2017	Select student samples for pilot
April—May 2017	Collect pilot data
End of May 2017	Deliver raw data to international sponsoring organization
July—August 2017	Review pilot data results
April—December 2017	Prepare data collection manuals, forms, assessment materials, questionnaires for field test
March—April 2017	Select school samples for field test
May—December 2017	Contact and gain cooperation of states, districts, and schools for field test
February—March 2018	Select student samples for main study
March 2018—April 2018	Collect field test data
May 15, 2018	Deliver raw data to international sponsoring organization
July 2018—August 2018	Review field test results
May 2018—December 2018	Prepare for the main study/recruit schools
March 2019—May 2019	Collect main study data
June 2020	Receive final data files from international sponsors
June 2020—December 2020	Produce reports

Note: Items in gray font have been concluded at the time of this submission, in June 2017.

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions to the certifications are requested.

1 The average hourly earnings of principals/education administrators in the May 2016 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $45.86, for school coordinators and teachers is $28.38, and of parents is $23.86. If mean hourly wage was not provided it was computed assuming 2,080 hours per year. The student wage is based on the federal minimum wage. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032), Education School Teachers, Except Special Education (25-2021), and all employees (00-0000); accessed on June 2, 2017.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Calvin Choi
File Modified	0000-00-00
File Created	2021-01-22

Part A TIMSS 2019 Main Study Recruitment & Field Test