Trends in International mathematics and science study (TIMSS 2019)
Field Test Recruitment and PILOT test
REQUEST FOR OMB Clearance
OMB# 1850-0695 v.8
Submitted by:
National Center for Education Statistics
U.S. Department of Education
Institute of Education Sciences
Washington, DC
August 2016
A.1 Importance of Information 3
A.2 Purposes and Uses of Data 3
A.3 Improved Information Technology (Reduction of Burden) 5
A.4 Efforts to Identify Duplication 5
A.5 Minimizing Burden for Small Entities 6
A.6 Frequency of Data Collection 6
A.8 Consultations outside NCES 6
A.9 Payments or Gifts to Respondents 6
A.10 Assurance of Confidentiality 7
A.13 Total Annual Cost Burden 9
A.14 Annualized Cost to Federal Government 10
A.15 Program Changes or Adjustments 10
A.16 Plans for Tabulation and Publication 10
A.17 Display OMB Expiration Date 11
A.18 Exceptions to Certification Statement 11
A: TIMSS 2019 Pilot and Field Test Recruitment Materials
B: TIMSS 2019 Pilot and Field Test Notification Letters and Supporting Materials
C: TIMSS 2019 Pilot Test Questionnaire Materials
The National Center for Education Statistics (NCES), within the U.S. Department of Education (ED), is requesting clearance for data collection materials and procedures for the 2019 Trends in International Mathematics and Science Study (TIMSS 2019) field test recruitment and pilot test, both due to begin in spring 2017. The Supporting Statements Part A and B in this submission are the same as those approved in August 2016 for TIMSS 2019 Pilot Test Recruitment (OMB# 1850-0695 v.7), with the following changes:
References to and burden for the pilot test school administrator (a.k.a., school) questionnaire have been removed because NCES decided that this questionnaire will not be fielded during the pilot test;
Total requested burden estimates in Part A were expanded to include burden for the pilot test data collection and field test recruitment; and
Sampling information for the TIMSS field test was added to Part B.
Appendix A provides the approved (OMB# 1850-0695 v.7) communication materials for the pilot test, and has been expanded to also provide field test recruitment materials (consisting of letters to state and district officials and school principals, text for a TIMSS brochure, Frequently Asked Questions, Summary of School Activities, and instructions for providing class and student lists). Parental notification materials and consent forms are provided in Appendix B, and the final versions of the pilot student questionnaires in Appendix C.
TIMSS is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years (1995, 1999, 2003, 2007, 2011, and 2015), with the next TIMSS assessment, TIMSS 2019, being the seventh iteration of the study. In TIMSS 2015, 48 countries and 6 other education systems participated at grade 4, and 38 countries and 6 other education systems participated at grade 8. The United States will participate in TIMSS 2019 to continue to monitor the progress of its students compared to that of other nations and to provide data on factors that may influence student achievement.
New in 2019, TIMSS will be a technology-based assessment conducted in an electronic format (referred to as “eTIMSS”) and administered using the eTIMSS Assessment Platform. There are two primary goals for the transition to a technology-based assessment: (1) maintaining continuity to measure trends with the past paper-and-pencil TIMSS assessments, and (2) developing relevant assessment innovations that take advantage of new technologies.
TIMSS is led by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the frameworks used to develop the assessment, the survey instruments, and the study timeline. IEA decides and agrees upon a common set of standards, procedures, and timelines for collecting and reporting data, all of which must be followed by all participating countries. As a result, TIMSS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., NCES conducts this study in collaboration with the IEA and a number of contractors (Westat, Avar Consulting (Avar), AIR, and Hager Sharp) to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in TIMSS is consistent with NCES’s mandate of acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9543].
Compared to previous assessment cycles, TIMSS 2019 differs in several ways:
Unlike TIMSS 2015, the IEA will not conduct TIMSS Advanced.
TIMSS 2019 will be a digitally-based assessment administered on the eTIMSS Assessment Platform.
TIMSS 2019 will administer a pilot study in 2017, followed by a field test in 2018.
The United States will participate in the 2017 pilot study to assist in the development of eTIMSS. The pilot is a very important part of the transition from a paper-based assessment to a technology-based assessment. The pilot is designed to provide information that can be used to reduce the data collection burden in 2019. The purpose of the pilot study is to (1) try out the new eTIMSS Assessment Platform,using the tablet player, data monitoring system, and online scoring; (2) conduct a mode effect study to examine the effect of administering the assessment on tablet versus paper; and (3) pilot newly-developed problem-solving and inquiry assessment items.
Data collection for the pilot will occur in April 2017. The United States plans to recruit 25 public schools at each grade (4 and 8), and assess 800 students at each of these grade levels. Student samples will be obtained by selecting two classes from each school (e.g., two mathematics classes for grade 8).
In preparation for the TIMSS 2019 main study, all countries are asked to implement a 2018 field test. The purpose of the TIMSS field test is to evaluate new assessment items and background questions to ensure practices that promote low exclusion rates, and to ensure that classroom sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively. Additionally, TIMSS staff will work to help respondents understand the study’s value relative to burden imposed, and to ensure a high level of school participation in the main study.
Because TIMSS is a collaborative effort among many parties, the United States must adhere to the international schedule set forth by the IEA, including the availability of final field test and main study plans as well as draft and final questionnaires. In order to meet the international data collection schedule for the spring 2017 pilot, recruitment activities must begin in October 2016. Recruitment for the field test will begin in May 2017, with recruitment for the main study in May of 2018 to align with recruitment for other NCES studies (e.g., the National Assessment of Education Progress, NAEP), and for schools to put the assessment on their calendars. Data collection for the field test will occur from March through April 2018. The U.S. TIMSS main study will be conducted from April through May, 2019. Due to the overlap in the timing of activities, this submission carries over the approved request for the 2017 pilot study recruitment (OMB# 1850-0695 v.7) and requests to conduct the pilot study data collection and field test recruitment.
In May 2017, NCES will submit a request to conduct the 2018 field test and recruit schools for TIMSS 2019 Main Study. In late 2018, NCES will submit another request, with a 30-day notice published in the federal register, which will include the final main study instruments for data collection in April-May, 2019. The main study questionnaires will be a subset of the field test instruments.
In addition to the field test, NCES may conduct cognitive labs at grades 4 and 8, and focus groups of school principals may be conducted to examine ways to increase participation. A separate OMB request will be submitted for these activities under NCES’s generic clearance for developmental studies (OMB# 1850-0803).
Benchmarking of U.S. student achievement against other countries continues to be of high interest to education policymakers, and informs policy discussions of economic competitiveness and workforce and post-secondary preparedness. TIMSS provides a unique opportunity to compare U.S. students‘ mathematics and science knowledge and skills at fourth and eighth grade with that of their peers in countries around the world. Science, technology, engineering, and mathematics (STEM) preparedness is key to economic improvement.
The continuation of U.S. participation allows for the study of past and current education policies that have shaped science and mathematics achievement over the past 24 years. Furthermore, participating countries are not only able to obtain information about students' knowledge and abilities in the specified subjects, but also about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement in the respective subject areas.
TIMSS complements what we learn from national assessments such as the National Assessment of Educational Progress (NAEP) by identifying the strengths and weaknesses of student science and mathematics achievement relative to participating countries around the world. It provides valuable benchmarking information about educational polices enacted in other countries and policies that could be applied to U.S. educational practices.
Based on earlier TIMSS data releases, it is likely that the results of TIMSS 2019 will draw great attention in the United States and elsewhere. It is therefore expected that TIMSS will contribute to ongoing national and international debates and efforts to improve mathematics and science learning and achievement.
TIMSS assesses mathematics and science knowledge and skills at grades 4 and 8. TIMSS is designed to align broadly with curricula in the participating countries. The results, therefore, suggest the degree to which students have learned concepts and skills likely to have been taught in school. TIMSS also collects background information on students, teachers, schools, curricula, and official education policies in order to allow cross-national comparison of educational contexts that may be related to student achievement.
Data compiled and collected from TIMSS 2019 allows for evidence-based decisions to be made for the purposes of educational improvement. Each successive participation in TIMSS provides trend information about student achievement in mathematics and science relative to other countries, as well as indicators that show how this achievement relates to demographic and curricular, school, teacher, and student factors that provide the educational context for achievement. This high quality, internationally comparative trend data provide key information to inform education policy discussions.
Through participation in TIMSS and other international assessment programs, NCES is able to provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance, and to suggest hypotheses about the relationship between student performance and factors that may influence performance as well as areas in which students have strengths or weaknesses. The international studies identify differences among countries over time in instructional practices, school policies, and opportunity-to-learn that informs discussions about how to organize instruction.
NCES’s mandate [Section 406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1)] specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the United States and in other nations." and the Educational Sciences Reform Act of 2002 (ESRA 2002: 20 U.S.C., Section 9543) specifies that NCES shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including acquiring and disseminating data on educational activities and student achievement in the United States compared with foreign nations. TIMSS is essential for any international perspective on students’ mathematics and science knowledge and skills, and U.S. participation in ICILS is aligned with both the national and international aspects of NCES' mission
The mathematics and science assessments at grades 4 and 8 are organized around a content dimension that specifies the subject matter to be assessed and a cognitive dimension that specifies the thinking processes to be assessed. The cognitive domains are the same in mathematics and science: knowing, applying, and reasoning. The TIMSS 2019 frameworks will be similar to 2015, but have been slightly revised or updated to provide more specificity for item writers, and to better reflect current curricula in participating countries. It is not anticipated that there will be any revisions to the content domains or cognitive domains, nor to the target percentages for the content domains or cognitive domains at either subject at either grade.
In fourth-grade mathematics the cognitive content domains include: number, geometric shapes and measures, and data display. More advanced content in these three domains are assessed in eighth grade, supplemented by a data and chance domain. TIMSS assesses a range of problem-solving situations within mathematics, with about two-thirds of the questions requiring students to use applying and reasoning skills.
In science at fourth grade, the content domains include: life science, physical science, and earth science. At eighth grade the content domains transition to a more discipline-based approach, reflecting the differences in instruction from elementary school. The content domains at eighth grade are: biology, chemistry, physics, and earth science. TIMSS 2019 will also measure science practices and science inquiry, reflecting recent emphasis on these skills in many countries’ curricula and content standards.
Assessment Mode
In TIMSS 2019 countries can choose to administer a digitally-based assessment, rather than the traditional paper-and-pencil mode of assessment. The United States will participate in digitally-based TIMSS. Studies will be conducted in the 2017, 2018, and 2019 to operationalize the new digital mode of assessment, as well as to analyze mode effects so that the important TIMSS trend reporting can be maintained.
The assessments will be conducted on tablets, such as the Microsoft Surface Pro, with a stylus and external keyboard.
Assessment Instruments
In order to minimize burden and to ensure broad subject-matter coverage, TIMSS will use a matrix sampling approach where the mathematics and science items at each grade level are organized into a set of test booklets, with each student taking only one booklet. Test items are either multiple-choice or constructed response items.
Questionnaires
The background questionnaires for TIMSS 2019 are being developed to address a background questions framework developed internationally. The United States will adapt the questions to fit the U.S. education context, including adding a few questions, such as about the race/ethnicity of students. Teacher and school questionnaires will be offered online, with a paper-and-pencil backup. Students will answer their questionnaires on the tablets, along with the cognitive assessment questions.
School Questionnaire. A representative from each participating school will be asked to provide information on mathematics and science resources, teacher availability and retention, principal leadership, school emphasis on academic success, school climate, and parental involvement in school activities. The TIMSS school questionnaire is expected to take 20 minutes to complete and will be offered online, with a paper-and-pencil backup. During the pilot study we will not be administering a school questionnaire.
Teacher Questionnaire. At grades 4 and 8, mathematics and science teachers of students in selected classes will be asked to complete a teacher questionnaire. Teacher questionnaires will include questions about teacher preparation and experience, mathematics and science topics taught, instructional resources and technology, instructional time, instructional engagement, classroom assessment, and technology resources and instruction in their classes. The teacher questionnaire is expected to take 30 minutes to complete and will be offered online, with a paper-and-pencil backup. We do not anticipate administering the teacher questionnaire during the pilot study.
Student Questionnaire. Student information will be collected about home resources, student motivation, self-concept, self-efficacy, and student characteristics such as gender and race/ethnicity. The student questionnaire is expected to take 30 minutes to complete and is administered after the cognitive assessment on the same digital device. During the pilot study we anticipate administering only an abbreviated student questionnaire that will take about 10 minutes to complete and will collect limited demographic and background information (see Appendix C).
The TIMSS 2019 design and data collection procedures are prescribed internationally including student assessments and questionnaires being administered on tablets. TIMSS will also administer paper-and-pencil versions of the assessment in the 2017 pilot test and to a subsample of students in the 2019 main study to study the mode effect of transitioning from paper to digital administration. Each participating nation is expected to adhere to the internationally prescribed design. In the United States, the school and teacher questionnaires will be made available to school administrators and teachers online as the main mode of administration, with a paper-and-pencil backup to facilitate user preference for participation.
A communication website, MyTIMSS USA, will be used during the 2017 pilot, 2018 field test, and 2019 main study in order to provide a simple, single source of information to engage sample schools and maintain high levels of their involvement. This portal will be used throughout the assessment cycle to inform schools of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather class and student lists from participating schools electronically using an adaptation of Westat’s secure E-filing process. E-filing is an electronic system for submitting lists of student information, including student background information in school records. Instructions to school coordinators on how to submit class and student lists are included in Appendix A. E-filing has been used successfully in NAEP for more than 10 years, and was used in TIMSS 2015 and the PISA 2012 and 2015 assessments. The E-filing system provides advantageous features such as efficiency and data quality checks. Schools will access the E-filing system through the MyTIMSS web site.
The eTIMSS assessments will be implemented using tablets carried into schools by the data collection staff so that TIMSS will not need to use school equipment. Schools can continue to use their own equipment for instruction and assessment, and TIMSS will not need to burden school or district IT staff to set up school equipment or take down firewalls. TIMSS does not require internet access during the assessment. Student data are uploaded to Westat secure servers after the assessment.
In the United States, mathematics and science achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971 through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. TIMSS 2019 does not duplicate these assessments.
TIMSS 2019 is part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent the U.S. participation in international studies involving approximately 60-65 countries. As part of international cooperative studies, the United States must collect the same information at the same time as the other participating nations for purposes of making both valid international comparisons with other countries and with the previous TIMSS data. While some studies in the United States collect similar, though not identical, types of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in TIMSS in that they do not allow for comparisons outside the United States. Furthermore, the data collected through TIMSS is based on unique frameworks that are not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the United States must agree to administer the same core instruments that are administered in all other participating countries. Because the items measuring mathematics and science achievement have been developed with intensive international coordination, any changes to the instruments require international coordination and approval.
No small entities are part of this sample. The school samples for TIMSS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. All school sizes are needed to ensure an appropriate representation of each type of school in the selected sample of schools. Burden will be minimized wherever possible. For example, schools will be selected so as to avoid overlap with other NCES assessments such as NAEP and PISA. In addition, contractor staff will conduct all test administrations, provide all equipment, and will assist with parental notification, sampling, and other tasks as much as possible within each school.
The pilot data collection is scheduled for April, 2017. The field test data collection is scheduled for March 1 through April 30, 2018. The full-scale data collection is scheduled for April through May, 2019. This schedule is prescribed by the international collective for TIMSS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries as well as to maintain trend lines.
None of the special circumstances identified in the Instructions for Supporting Statement apply to the TIMSS study.
Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An International Steering Committee has general oversight of the study and each National Research Coordinator participates in extensive discussions concerning the projects, usually with advice from national subject matter and testing experts. In addition, the IEA convened separate panels of mathematics and science experts from around the world to develop cognitive items.
The majority of the consultations (outside NCES) have involved the TIMSS International Study Center at Boston College in the United States. Key to these ongoing consultations are: Dirk Hastedt (executive director of the IEA); Michael Martin, Ina V.S. Mullis, Victoria Centurino, and Kerry Cotter, all of whom have extensive experience in developing and operating international education surveys (especially related to TIMSS).
In order to achieve acceptable school response rates in international studies, schools in the U.S. are usually offered $200 to thank them for their participation and the time they invest in and the space they make available for the international assessments. High response rates are required by both IEA and NCES, and are difficult to achieve in school-based studies. The U.S. has historically had difficulties in achieving sufficient participation levels. Based on incentives provided in past administrations of TIMSS and currently offered in other international assessments, schools will be offered $200 for their time in the field test and the main study. In the 2017 eTIMSS pilot, students will take both the paper-and-pencil version and an eTIMSS version of the assessment. Student time will be twice what it typically is, and will involve more assistance from the school staff in order to administer either two sessions in one day, or over two days. Therefore, schools participating in the eTIMSS pilot will be offered $400 for their time, help, and participation.
The school staff serving as School Coordinators will receive $100 for their time and effort. The School Coordinator serves a critical role in data collection, functioning as the central school contact and facilitating arrangements for the assessments. They are asked to file class and student listing forms; arrange the date, time and space for the assessment; and disseminate information to parents and students.
For the first time, TIMSS will offer $20 to teachers who complete the teacher questionnaires. Both PISA and PIRLS (in 2016) have offered this level of incentive to teachers who complete a 30-minute questionnaire. In the past, field staff and Westat home office staff have spent considerable time tracking down teachers after the assessment to obtain a reasonable response rate. However, during the pilot study, we do not anticipate administering the teacher questionnaire and thus incentivizing teachers.
Consistent with prior administrations of TIMSS, as a token of appreciation for their participation, students will receive a small gift valued at approximately $4. In TIMSS 2015, each participating student received a small flashlight that could be clipped with an attached karabiner to a backpack or belt loop. Students will also receive a certificate with their name thanking them for participating and representing the United States in TIMSS. In schools where it is permitted, eighth-grade students participating in TIMSS may also receive a certificate from the U.S. Department of Education for four hours of volunteer service. Additionally, some schools also offer recognition parties with pizza or other treats for students who participate; however these are not reimbursed by NCES or the contractor.
The laws pertaining to the collection and use of personally identifiable information are clearly communicated in correspondence with states, districts, schools, teachers, students, and parents. Letters and information materials will be sent to parents and school administrators describing the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential (see copies in appendix A):
NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By law, the data provided [by you, schools, staff, and students] may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. § 9573).
The following statement will appear on the login page for eTIMSS, MyTIMSS, and the front cover of the printed questionnaires (the phrase “search existing data resources, gather the data needed” will not be included on the student questionnaire):
The National Center for Education Statistics (NCES), within the U.S. Department of Education, conducts TIMSS in the United States as authorized by the Education Sciences Reform Act of 2002 (20 U.S.C., § 9543). Your responses are protected by federal statute (20 U.S.C., § 9573) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0695. The time required to complete this information collection is estimated to average [XX] minutes per [respondent type], including the time to review instructions [, search existing data resources, gather the data needed,] and complete and review the information collection. If you have any comments or concerns regarding the accuracy of the time estimate(s), suggestions for improving the form, or questions about the status of your individual submission of this form, write directly to: Trends in Mathematics and Science Study (TIMSS), National Center for Education Statistics, PCP, 550 12th St., SW, 4th floor, Washington, DC 20202.
OMB No. 1850-0695, Approval Expires xx/xx/2019.
The TIMSS 2019 confidentiality plan includes signing confidentiality agreements and notarized nondisclosure affidavits by all contractor and subcontractor personnel and field workers who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility. Data files, accompanying software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included on any data file. In eTIMSS, students log into the automated assessment using non-identifying ID’s and thus the resulting data are collected and stored with only the non-identifying TIMSS assigned ID. The data are collected from the tablets using a process that encrypts the data, and uploads the data onto a Westat secure FTP site.
NCES understands the legal and ethical need to protect the privacy of the TIMSS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. The contractor will conduct a thorough disclosure analysis of the TIMSS 2019 data when preparing the data files for use by researchers, in compliance with 20 U.S.C., § 9573. Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including swapping data and omitting key identification variables (i.e., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or ZIP Code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files.
The questionnaires do not have items considered to be of a sensitive nature.
This request is to conduct TIMSS 2019 field test recruitment and pilot test. Due to the overlap in the timing of activities, this submission also carries over the approved request and burden for the 2017 pilot study recruitment (OMB# 1850-0695 v.7). Therefore, the burden estimates include burden for (1) contacting states, districts, schools, and parents in order to begin recruitment for the eTIMSS pilot and field test, including (a) sending recruitment letters to districts and schools selected for each study, (b) contacting and seeking research approvals from special handling districts, where applicable, and (c) notifying parents of sampled students about their participation in either study; and (2) collecting pilot test data. The estimated burden for these efforts is shown in Table A.1.
The district and school contact letters for the pilot and field test are assumed to impose small burden on all contacted parties, both those that refuse and those that agree to participate in the TIMSS pilot or field test. Thus the burden hours for this activity are based on the total number of districts and schools contacted rather than the total number agreeing to participate.
The special handling districts are those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous TIMSS data collections, we estimate that there may be up to 10 special handling districts in the sample. Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. This operation should begin in fall 2016 to allow sufficient time for special handling districts’ review processes. We will begin contacting these districts upon receiving OMB’s approval, and continue to work with them until we receive a final response from each district (approval or denial of request) up until March 31, 2017 for the pilot and March 1, 2018 for the field test.
The total district and school response burden estimate for the field test recruitment and pilot test is based on 10 minutes for districts to read materials and respond, and 20 minutes for schools to read materials and respond. The total response burden estimate for IRB approvals is based on 120 minutes for staff approval and 60 minutes for panel approval. The total response burden for parental notification is based on 10 minutes for reading and reviewing recruitment materials and notification forms. The total response burden for the pilot data collection is based on 10 minutes for students to complete a short student questionnaire, and for school personnel to complete pre-assessment activities of 90 minutes for a school administrator and 240 minutes for a school coordinator. Based on the estimated hourly rates for principals/administrators and parents of $44.68 and $22.71, respectively1, and the federal minimum wage of $7.25 as the hourly rate for the students, and based on the estimated total of 2,190 burden hours for TIMSS 2019 field test school recruitment and pilot test, the estimated response burden time cost is $41,099.
Table A.1. Burden estimates for TIMSS 2019 pilot data collection in 2017 and for TIMSS 2019 pilot and field test recruitment, in 2016 and 2017, respectively, for grades 4 and 8.
Activity |
Sample size |
Expected response rate |
Number of respondents |
Number of responses |
Per respondent (minutes) |
Total burden (hours) |
Pilot Recruitment |
|
|
|
|
|
|
Contacting Districts |
72 |
1.00 |
72 |
72 |
10 |
12 |
Contacting Schools |
72 |
1.00 |
72 |
72 |
20 |
24 |
District IRB Staff Study Approval |
10 |
1.00 |
10 |
10 |
120 |
20 |
District IRB Panel Study Approval |
60 |
1.00 |
60 |
60 |
60 |
60 |
Parental notification |
1,250 |
1.00 |
1,250 |
1,250 |
10 |
209 |
Pilot Recruitment Burden |
|
|
1,464 |
1,464 |
|
325 |
Pilot Data collection instrument |
|
|
|
|
|
|
Students |
|
|
|
|
|
|
Grade 4 |
|
|
|
|
|
|
Directions* |
1,250 |
0.80 |
1,000 |
1,000 |
20 |
334 |
Assessment* |
1,250 |
0.80 |
1,000 |
1,000 |
144 |
2,400 |
Student Questionnaire |
1,250 |
0.80 |
1,000 |
1,000 |
10 |
167 |
Grade 8 |
|
|
|
|
|
|
Directions* |
1,250 |
0.80 |
1,000 |
1,000 |
20 |
334 |
Assessment* |
1,250 |
0.80 |
1,000 |
1,000 |
180 |
3,000 |
Student Questionnaire |
1,250 |
0.80 |
1,000 |
1,000 |
10 |
167 |
Total Student Burden Pilot |
|
|
2,000 |
4,000 |
|
1,002 |
Pre-Assessment Activity |
|
|
|
|
|
|
School Administrator |
72 |
0.69 |
50 |
50 |
90 |
75 |
School Coordinator (Grade 4 and 8) |
72 |
0.69 |
50 |
50 |
240 |
200 |
Total School Burden Pilot |
|
|
100 |
100 |
|
275 |
Total Burden Pilot |
|
|
3,564 |
5,564 |
|
1,602 |
Field Test Recruitment |
|
|
|
|
|
|
Contacting Districts |
80 |
1.00 |
80 |
80 |
10 |
14 |
Contacting Schools |
80 |
1.00 |
80 |
80 |
20 |
27 |
District IRB Staff Study Approval |
10 |
1.00 |
10 |
10 |
120 |
20 |
District IRB Panel Study Approval |
60 |
1.00 |
60 |
60 |
60 |
60 |
Parental notification |
2,800 |
1.00 |
2,800 |
2,800 |
10 |
467 |
Field Test Recruitment Burden |
|
|
3,030 |
3,030 |
|
588 |
Field Test Data collection instrument |
|
|
|
|
|
|
Student |
|
|
|
|
|
|
Grade 4 |
|
|
|
|
|
|
Directions* |
1,550 |
0.93 |
1,442 |
1,442 |
20 |
481 |
Assessment* |
1,550 |
0.93 |
1,442 |
1,442 |
144 |
3,461 |
Student Questionnaire |
1,550 |
0.93 |
1,442 |
1,442 |
50 |
1,202 |
Grade 8 |
|
|
|
|
|
|
Directions* |
1,550 |
0.93 |
1,442 |
1,442 |
20 |
481 |
Assessment* |
1,550 |
0.93 |
1,442 |
1,442 |
180 |
4,326 |
Student Questionnaire |
1,550 |
0.93 |
1,442 |
1,442 |
50 |
1,202 |
Total Student Burden Field Test |
|
|
2,884 |
5,768 |
|
3,366 |
School Staff (Grades 4 and 8) |
|
|
|
|
|
|
School Administrator |
80 |
0.95 |
76 |
76 |
30 |
38 |
Teacher |
320 |
0.95 |
304 |
304 |
30 |
152 |
School Coordinator (Grade 4 and 8) |
80 |
1.00 |
80 |
80 |
240 |
320 |
Total School Burden Field Test |
|
|
460 |
460 |
|
510 |
Total Burden Field Test |
|
|
6,374 |
9,258 |
|
4,464 |
Total burden requested in this submission |
|
|
6,594 |
8,594 |
|
2,190 |
There are no additional costs to respond beyond the time to respond.
The cost to the federal government for conducting initial phases of TIMSS 2019, including all 2017 pilot operations (preparation, recruitment, pilot data collection, and scoring), and recruitment for the 2018 field test is estimated to be $1,361,821 over a 2-year period. This figure includes all direct and indirect costs.
The apparent increase in burden from last approval is due to the fact that the last request was only to recruit schools and communicate with districts and parents in preparation for the 2017 eTIMSS pilot study, while this request includes the additional burden to conduct the TIMSS 2019 pilot test and to recruit schools and communicate with districts and parents in preparation for the 2018 TIMSS 2019 field test.
With respect to the previous administration of Main Study TIMSS, TIMSS:2019 differs from TIMSS:2015 in the following ways:
No federally funded state benchmarking;
Transition to a digitally-based assessment in 2019;
Addition of a pilot study preceding the field test; and
TIMSS Advanced will not be administered in 2019.
The TIMSS pilot and field test are designed to provide a statistical review of the performance of items on the cognitive assessment and questionnaires in preparation for the main study data collection. They will also provide valuable experience in administering eTIMSS on tablets and the performance of TIMSS items in a digitally-based environment. Corresponding paper-and-pencil assessments in 2017 and 2019 will provide information for evaluating the mode effect of the transition from paper to eTIMSS.
Based on the data collected in the main study, the TIMSS International Study Center will prepare separate reports for mathematics and science at grades 4 and 8. These reports will be released by December 2020. As has been customary, NCES will also release a report for each study at the same time as the international reports are released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of either dataset. Examples of past reports on TIMSS can be found at http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2013009. In the spring of 2021, the International Study Center will also prepare technical reports for TIMSS 2019, describing the design and development of the assessments as well as the scaling procedures, weighting procedures, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for each study.
Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of TIMSS 2019 results in the United States are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:
Dates |
Activity |
April—December 2016 |
Prepare data collection manuals, forms, assessment materials, and questionnaires for pilot |
July—August 2016 |
Select school samples for pilot |
October 2016—January 2017 |
Contact and gain cooperation of states, districts, and schools for pilot |
February—March 2017 |
Select student samples for pilot |
April—May2017 |
Collect pilot data |
End of May 2017 |
Deliver raw data to international sponsoring organization |
July—August 2017 |
Review pilot data results |
April—December 2017 |
Prepare data collection manuals, forms, assessment materials, questionnaires for field test |
March—April 2017 |
Select school samples for field test |
May—December 2017 |
Contact and gain cooperation of states, districts, and schools for field test |
February—March 2018 |
Select student samples for field test |
March 2018—April 2018 |
Collect field test data |
May 15, 2018 |
Deliver raw data to international sponsoring organization |
July 2018—August 2018 |
Review field test results |
May 2018—December 2018 |
Prepare for the main study/recruit schools |
March 2019—May 2019 |
Collect main study data |
June 2020 |
Receive final data files from international sponsors |
June 2020—December 2020 |
Produce reports |
The OMB expiration date will be displayed on all data collection materials.
No exceptions to the certifications are requested.
1 The average hourly earnings of principals/education administrators in the May 2015 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $44.68, and of parents is $22.71. If mean hourly wage was not provided it was computed assuming 2,080 hours per year. The student wage is based on the federal minimum wage. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032) and all employees (00-0000); accessed on April 5, 2016.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Calvin Choi |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |