OMB83C Change Memo v13

DATE: July 17, 2018

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, NCES

FROM: Lydia Malley, NCES

Re: Trends in International Mathematics and Science Study (TIMSS 2019) Main Study Recruitment Change Request (OMB# 1850-0695 v.13)

The Trends in International Mathematics and Science Study (TIMSS) is an international assessment of fourth and eighth grade students’ achievement in mathematics and science. Since its inception in 1995, TIMSS has continued to assess students every 4 years, with the next TIMSS assessment, TIMSS 2019, being the seventh iteration of the study. TIMSS provides a comparison of U.S. student performance with those of their international peers in mathematics and science at grades 4 and 8. TIMSS is coordinated by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that creates the assessment framework, assessments, and background questionnaires and provides procedures and technical standards which all countries must follow. In the U.S., the National Center for Education Statistics (NCES) conducts TIMSS. In preparation for the TIMSS 2019 main study, NCES conducted a field test in 2018 to evaluate new assessment items and background questions, to ensure practices that promote low exclusion rates, and to ensure that classroom and student sampling procedures proposed for the main study are successful. The U.S. TIMSS 2019 field test recruitment began in May 2017, with data collection conducted from March through April 2018. The U.S. TIMSS 2019 main study recruitment began in May 2018, with data collection to be conducted from March through May 2019. The request for the TIMSS 2019 Main Study recruitment & Field Test was approved in July 2017 with the latest change request approved in January 2018 (OMB# 1850-0695 v.10-12).

This request is to: (a) provide the final versions of the recruitment materials that will be used during the later portion of the TIMSS 2019 Main Study, (b) implement the strategy used in other international studies of a second-tier school incentive designed to help meet the response rates required for inclusion in international comparisons; (c) update the main study burden estimates; and (d) update the cost to the federal government for this study. The revisions made to the approved TIMSS 2019 clearance documents are detailed below. Text in red font denotes newly added text, crossed out red font denotes deleted text, and blue font reflects notes about the edits.

Revisions to Supporting Statement Part A

Preface

(…) This request is to provide additional recruitment materials that will be used during the later portion of field test recruitment and throughout the main study, and to provide the final international versions of the TIMSS 2019 field test questionnaires including the proposed U.S. adaptations, and to implement the strategy used in other international studies of a second-tier incentive designed to help meet the response rates required for inclusion in international comparisons.

(…) In this submission, Appendix A-B includes newly added communication materials for the later portion of field test recruitment and for the main study. U.S. versions of the TIMSS 2019 field test questionnaires are provided in Appendix C2, with a summary of changes to the instruments from TIMSS 2015 Main Study to the TIMSS 2019 Field Test provided in Appendix C1. International and U.S. versions of the TIMSS 2019 main study questionnaires will be provid­ed in a subsequent submission in the fall of 2018.

(…) In addition to the field test, NCES may conduct focus groups of school principals to examine ways to increase participation or field trials of electronic instruments to test the electronic systems. In such case, a separate OMB request will be submitted for these activities under NCES’s generic clearance for developmental studies (OMB# 1850-0803).

A.2 Purposes and Uses of Data

School Questionnaire: The time the TIMSS school questionnaire is expected to take has been updated from 20 minutes to 30 minutes, per international guidelines. Where needed, this has been also updated in Appendix communication materials for TIMSS Main Study. The 30 minutes was already reflected in the burden table.

(…) The TIMSS school questionnaire is expected to take 20 30 minutes to complete and will be offered online, with a paper-and-pencil backup. The school questionnaire was not administered during the pilot study.

A.5 Minimizing Burden for Small Entities

(…) For example, schools will be selected so as to avoid as much as possible overlap with other NCES assessments such as NAEP and PISA. In addition, contractor staff will conduct all test administrations, provide all equipment, and will assist with parental notification, sampling, and other tasks as much as possible within each school.

A.9 Payments or Gifts to Respondents

(…) To address challenges that may be encountered with securing school cooperation for TIMSS 2019, we propose a second-tier incentive which will allow us to offer up to $800 to schools that are historically very difficult to recruit. This second-tier incentive will be offered only to:

• Private schools in the original sample or substitute private schools. In ICILS 2018, only 40% of original private schools participated, with the overall rate of about 57% after substitutes were added.

• Public schools that are selected for and participate in NAEP 2019 and are also selected for TIMSS 2019. Typically these schools participate only in NAEP because it is required, and refuse participation in TIMSS. There are about 40 such schools nationwide.

• All substitute schools. Typically, refusals by original schools occur in the mid to late fall prior to the assessment in the spring, at which point substitute schools are recruited. Recruitment of substitutes is typically very challenging (less than 25% participate) because recruitment is late, after the school year has begun and school calendars are set (also, some states refuse to recruit schools that late in the school year).

A similar second-tier recruitment strategy has been used in other international studies conducted by NCES. Most recently, in ICILS 2018, we began offering the second-tier incentive very late, in the middle of data collection, and attempted to turn around schools that had already refused, as well as newly activated substitutes and schools that were not doing any work to prepare for the assessment. We were able to successfully recruit about 20% of these schools even with this late start, which brought us much closer to our target recruitment percentage. The extra incentive strategy should be even more successful in TIMSS if we begin it during the fall, well before data collection begins.

In order to enhance our teacher recruitment and participation strategy we will deliver the questionnaire via email rather than a Login card and will send a reminder email and electronic gift card. The previously approved $20 incentive will be distributed through Amazon gift cards, as has been done in ICILS 2018.

(…) To encourage their participation, TIMSS will offer for the first time $20 to teachers who complete the teacher questionnaires. Both PISA and PIRLS (in 2016) have offered this level of incentive to teachers who completed a 30-minute questionnaire. In the past, field staff and Westat home office staff have spent considerable time tracking down teachers after the assessment to obtain a reasonable response rate. The teacher questionnaire was not administered during the pilot test and thus no teacher incentive was used. In order to avoid sending 5 to 10 checks to the school for the school coordinator to distribute to teachers who complete the questionnaire, electronic Amazon gift cards in the amount of $20 will be used. Amazon gift cards were selected because they have no associated fees, unlike other cash card programs. Teacher email addresses will be collected prior to the assessment, and will be used to electronically deliver their login credentials (i.e., user ID and password) for accessing the teacher questionnaire on the IEA Online Survey System. The teacher email addresses will also be used to send teachers their incentive (i.e., Amazon gift code) after the questionnaire has been completed, and to send periodic reminders to teachers who have not yet completed their questionnaires.

To encourage active parental consent forms to be returned for schools that require them, we added:

(…) In schools where it is permitted, eighth-grade students participating in TIMSS may also receive a certificate from the U.S. Department of Education for four hours of volunteer service. Additionally, some schools also offer recognition parties with pizza or other treats for students who participate; however these are not reimbursed by NCES or the contractor. In districts or schools that require active parental consent, which historically has been difficult to collect, we will offer a small party with refreshments for students who bring in their parental consent forms.

A.12 Estimates of Burden

The following was updated using the May 2017 National Occupational and Employment Wage Estimates:

(…) Based on the estimated hourly rates for principals/administrators, school coordinators, teachers, and parents of $45.86, $23.38, $23.38 $47.81, $29.25, $29.25, and $23.86 24.34, respectively¹, and the federal minimum wage of $7.25 as the hourly rate for the students, and based on the estimated total of 10.974 941 burden hours for TIMSS 2019 main study recruitment and field test recruitment and data collection, the estimated respondent burden time cost is $217,348 223,400.

When an originally sampled school decides in final stages of recruitment not to participate, recruitment for a replacement school for that original school may begin. Replacement schools are contacted only after an original school’s final refusal, and only in order to meet satisfactory participation rates. The burden table has been updated to account for replacement schools. Our expected response rate among original schools has declined and we are planning to activate up to 201 replacement schools, now shown in the burden table.

Table A.1. Burden estimates for TIMSS 2019 field test data collection in 2018 and for TIMSS 2019 field test and main study recruitment, in 2017 and 2018, respectively, for grades 4 and 8.

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Per respondent (minutes)	Total burden (hours)
Field Test Recruitment
Contacting Districts	80	1.00	80	80	10	14
Contacting Schools	80	1.00	80	80	20	27
District IRB Staff Study Approval	10	1.00	10	10	120	20
District IRB Panel Study Approval	60	1.00	60	60	60	60
Parental notification	2,800	1.00	2,800	2,800	10	467
Field Test Recruitment Burden			3,030	3,030		588
Field Test Data Collection
Student
Grade 4
Directions	1,550	0.93	1,442	1,442	20	481
Assessment	1,550	0.93	1,442	1,442	72	1,731
Student Questionnaire	1,550	0.93	1,442	1,442	50	1,202
Grade 8
Directions	1,550	0.93	1,442	1,442	20	481
Assessment	1,550	0.93	1,442	1,442	90	2,163
Student Questionnaire	1,550	0.93	1,442	1,442	50	1,202
Total Student Burden Field Test			2,884	5,768		3,366
School Staff (Grades 4 and 8)
School Administrator	80	0.95	76	76	30	38
Teacher	320	0.95	304	304	30	152
School Coordinator (Grade 4 & 8)	80	1.00	80	80	240	320
Total School Burden Field Test			460	460		510
Field Test Data Collection Burden			6,374	9,258		4,464
Main Study Recruitment
Contacting DistrictsDistrict Recruitment	670	1.00	670	670	10	112
ContactingSchool Recruitment (Original Schools)	670	1.00 0.70*	670 469	670 469	20	224 157
School Recruitment (Replacement Schools)	201	0.50*	101	101	20	34
District IRB Staff Study Approval	84	1.00	84	84	120	168
District IRB Panel Study Approval	504	1.00	504	504	60	504
Parental notification	29,480	1.00	29,480	29,480	10	4,914
Main Study Recruitment Burden			31,408308	31,408308		5,922889
Main Study Data Collection
Student
Grade 4
Directions	14,740	0.9396	13,709 14,151	13,709 14,151	20	4,570717
Assessment	14,740	0.9396	13,709 14,151	13,709 14,151	72	16,451982
Student Questionnaire	14,740	0.9396	13,709 14,151	13,709 14,151	50	11,425793
Grade 8
Directions	14,740	0.9396	13,709 14,151	13,709 14,151	20	4,570717
Assessment	14,740	0.9396	13,709 14,151	13,709 14,151	90	20,564 21,227
Student Questionnaire	14,740	0.9396	13,709 14,151	13,709 14,151	50	11,425793
Total Student Burden Main Study			27,418 28,302	54,836 56,604		31,990 33,020
School Staff (Grades 4 and 8)
School Administrator	670	0.95	637	637	30	319
Teacher	2,680	0.95	2,546	2,546	30	1,273
School Coordinator (Grade 4 & 8)	670	1.00	670	670	240	2,680
Total School Burden Main Study			3,853	3,853		4,272
Main Study Data Collection Burden			62,679 63,463	90,097 91,765		42,184 43,181
Total burden requested in this submission			40,812 40,712	43,696 43,596		10,974941

Note: Total burden requested in this submission includes burden associated with field test data collection and with recruiting schools and contacting districts and parents for the TIMSS 2019 field test and main study (items in black font). Cells showing burden that was approved by OMB for the 2018 field test recruitment (OMB# 1850-0695 v.8-9) are shaded in light gray and included in the requested burden totals. Burden that will be requested for the future TIMSS 2019 main study data collection or that is estimated for the field test cognitive assessments, which are not subject to PRA, are demarcated in gray font, are presented here for information only purposes, and are not included in burden totals.

* Satisfactory sampling participation rate includes a final unweighted school response rate of at least 50% of original schools and at least 85% of original plus replacement schools, with original school sample as the denominator.

A.14 Annualized Cost to Federal Government

Recruitment cost has increased as follows to account for the increased efforts and incentive costs of the second-tier incentive strategy:

The cost to the federal government for conducting initial phases of TIMSS 2019, including 2018 field test operations (preparations, data collection, and scoring) and 2019 main study recruitment is estimated to be $1,930,283 2,098,883 over a 2-year period (see table breakdown below). These figures include all direct and indirect costs.

Components with breakdown	Estimated costs
PILOT (2017)
Recruitment	95,000
Preparations (e.g., adapting instruments, sampling)	177,851
Data collection, scoring, and coding	323,822
FIELD TEST (2018)
Recruitment	149,885
Preparations (e.g., adapting instruments, sampling)	409,276
Data collection, scoring, and coding	949,635
MAIN STUDY (2019)
Recruitment	571,372 739,972
Preparations (e.g., adapting instruments, sampling)	53,278
Data collection, scoring, and coding	4,961,896
Current package components	$1,930,283 2,098,883
Grand total	$7,692,015860,615

Revisions to Appendix A (Recruitment Materials)

To reflect the heightened need for their participation and the higher second-tier incentives being offered, two letters were amended with text in purple font to be used with the second-tier schools (the approved text for first-tier schools is denoted in red below):

School District TIMSS 2019 Main Study Letter [First-Tier/Second-Tier] (p.78):

[Selected schools are notified in advance so that principals can place the assessment date on their calendars and incorporate TIMSS into the planned school program. I am writing to ask for your support of the participation of those selected schools. Most selected students will take the TIMSS assessment on supplied tablets. Some students will take a paper and pencil assessment in order to bridge eTIMSS to previous paper versions of TIMSS. Participating schools will receive $200, and each school’s TIMSS school coordinator (the school staff person designated to work with TIMSS staff) will receive $100 as a thank you for his or her time and effort./Selected schools are notified so that principals can place the assessment date on their calendars. I am writing to ask your district to support the participation of those selected schools. We understand that we are asking schools to participate in TIMSS later in the school year when school calendars are set <or> in addition to participating in the National Assessment of Educational Progress (NAEP) and therefore are offering each school <up to $800> as a thank you for participation. In addition, each school’s TIMSS school coordinator (the school staff person designated to work with TIMSS staff) will receive $100 as a thank you for his or her time and effort.]

School TIMSS 2019 Main Study Letter [First-Tier/Second-Tier] (p.79):

[As a token of appreciation for participating in TIMSS, your school will receive $200. / We are notifying your school after the school year has already begun, and recognize that school calendars are set <or> We understand that your school has also been selected for the National Assessment of Education Progress (NAEP) and are offering a special indication of appreciation for your school’s time. If you participate, we will provide your school a thank you of <up to $800>.]

Where relevant we reflected that the teacher incentive will provided via an Amazon gift card, for example:

“You will receive a an Amazon $20 gift card within a few weeks days of completing the questionnaire as a token of our appreciation.”

We updated the Instructions for Submitting Class Lists – Grade 4 (pp. 102-104 and screenshot on p. 109) & Grade 8 (pp.112-114 and screenshot on p. 119) for the following reasons:

1. To ask for teacher emails in class list submission

2. To modify list submission instructions by asking for student lists in selected classes only

3. To update screenshots by asking for teacher emails on the class list pages

4. To update tips found in class list submission instructions

Lastly, we added the following materials to enhance TIMSS Main Study teacher participation strategy:

• Teacher Questionnaire Selection Email to replace Teacher Questionnaire Login Card (p.132): we will deliver the teacher questionnaire via email rather than through a login card.

• Teacher Questionnaire Reminder Email (p.133): to remind teachers to complete the questionnaire.

• Amazon Gift Card Code Email (p.134): to distribute the Amazon Gift Cards to teachers upon completion of the questionnaire.

1 The average hourly earnings of principals/education administrators in the May 2016 2017 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $45.86 47.81, for school coordinators and teachers is $28.38 29.25, and of parents is $23.86 24.34. If mean hourly wage was not provided it was computed assuming 2,080 hours per year. The student wage is based on the federal minimum wage. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032), Education School Teachers, Except Special Education (25-2021), and all employees (00-0000); accessed on June 2, 2017 July16, 2018.