Update Memo

MEMORANDUM OMB # 1850-0803 v. 155

DATE: April 15, 2016

TO: Robert Sivinski, Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patricia Etienne, National Center for Education Statistics

THROUGH: Kashka Kubzdela, National Center for Education Statistics

SUBJECT: National Assessment of Educational Progress (NAEP) Pretesting of Survey and Cognitive Items for Pilot in 2017 and 2018 Update [Revision to a previously approved package (1850-0803 v.146)]

This request is to increase burden for the NAEP pretesting of cognitive items for pilot in 2018 (originally approved in October 2015, OMB #01850-0803 v. 146). Since the original submission, the scope of the grade 12 pretesting activities has increased. The original ICR package submission included pretesting activities for cognitive discrete items and has now been updated to also include pretesting activities for scenario-based tasks (SBTs) at grade 12. Specifically, we request to increase the burden to allow 50, 14, and 100 grade 12 students to participate in playtesting, cognitive interviews, and tryouts (respectively) of the SBTs. In association with this increase in burden, we request to adjust the timeframe and budget of the pretesting. Other than these specific changes, the objectives and scope of these pretesting activities remain the same as requested in the original submission.

The tables below (numbered as in Volume I) detail the requested changes to the estimated respondent burden (Tables 4-7), cost to federal government (Table 8), and the study schedule (Table 9). Cells highlighted in yellow pinpoint changes.

Table 4. Hourly Burden for Reading and Mathematics Cognitive Items Playtesting

	Original Estimates			Revised Estimates
Respondent	Hours per respondent	Number of respondents	Total hours	Hours per respondent	Number of respondents	Total hours
Student Recruitment via Teachers and Staff
Initial contact with staff: e-mail, flyer distribution, and planning	0.33	8	3	0.33	15	5
Sub-Total		8	3		15	5
Parent or Legal Guardian, and Student (18 or older)
Flyer and consent form review	0.08	160	13	0.08	294	24
Consent form completion and return	0.13	80*	11	0.13	147*	20
Confirmation to parent via email or letter	0.05	60*	3	0.05	110*	6
Sub-Total		160	27		294	50
Participation (Playtesting)											309	55
Grade 4	1	20	20	1	20	20
Grade 8	1.5	20	30	1.5	20	30
Grade 12	1.5	20	30	1.5	70	105
Sub-Total		60	80		110	155
Total Burden		228	110		419	210

Note: All totals have been rounded. In addition, some totals may differ slightly from sum of subtotals, due to rounding.

* Subset of initial contact group (total number of responses = 368 [original]; 676 [revised])

Table 5. Hourly Burden for Reading and Mathematics Cognitive Items Cognitive Interviews

	Original Estimates			Revised Estimates
Respondent	Hours per respondent	Number of respondents	Total hours	Hours per respondent	Number of respondents	Total hours
Student Recruitment via Teachers and Staff
Initial contact with staff: e-mail or phone, flyer distribution, and planning	0.33	12	4	0.33	14	5
Sub-Total		12	4		14	5
Parent or Legal Guardian, and Student (18 or older)
Initial flyer review or phone screener	0.08	224	18	0.08	262	21
Consent form completion and return	0.13	112*	15	0.13	131*	18
Confirmation to via email, letter, or phone	0.05	84*	4	0.05	98*	5
Sub-Total		224	37		262	44
Participation (Cognitive Interviews)
Grade 4	1	28	28	1	28	28
Grade 8	1.5	28	42	1.5	28	42
Grade 12	1.5	28	42	1.5	42	63
Sub-Total		84	112		98	133
Total Burden		320	153		374	182

Note: All totals have been rounded. In addition, some totals may differ slightly from sum of subtotals, due to rounding.

* Subset of initial contact group (total number of responses = 516 [original]; 603 [revised])

Table 6. Hourly Burden for Reading and Mathematics Cognitive Items Tryouts

	Original Estimates			Revised Estimates
Respondent	Hours per respondent	Number of respondents	Total hours	Hours per respondent	Number of respondents	Total hours
Student Recruitment via Teachers and Staff
Initial contact with staff: e-mail or phone, flyer distribution, and planning	0.33	40	13	0.33	54	18
Sub-Total		40	13		54	18
Parent or Legal Guardian, and Student (18 or older)
Initial flyer review or phone screener	0.08	800	64	0.08	1,068	86
Consent form completion and return	0.13	400*	52	0.13	534*	70
Confirmation to via email, letter, or phone	0.05	300*	15	0.05	400*	20
Sub-Total		800	131		1,068	176
Recruitment Totals		840	145		1,122	194
Participation (Tryouts)
Grade 4	1	100	100	1	100	100
Grade 8	1	100	100	1	100	100
Grade 12	1	100	100	1	200	200
Interview Totals		300	300		400	400
Total Burden		1,140	444		1,522	594

Note: All totals have been rounded. In addition, some totals may differ slightly from sum of subtotals, due to rounding.

* Subset of initial contact group (total number of responses = 1,840 [original]; 2,456 [revised])

Table 7. Combined Burden for Pretesting Activities

	Original Estimates			Revised Estimates
	Number of respondents	Number of responses	Burden Hours	Number of respondents	Number of responses	Burden Hours
Survey Questions
Total Cognitive Interview Burden	244	470	119	244	470	119
Cognitive Items
Total Playtesting Burden	228	368	110	419	676	210
Total Cognitive Interview Burden	320	516	153	374	603	182
Total Tryout Burden	1,140	1,840	444	1,522	2,456	594
Overall Totals	1,932	3,194	826	2,559	4,205	1,105

Table 8. Estimate of Costs

Activity	Provider	Original Estimated Cost	Revised Estimated Cost
Design, preparation, analysis, and reporting for survey questionnaire cognitive interviews Preparation and conduct for survey questionnaire cognitive interviews (including recruitment, incentive costs, data collection and documentation)	ETS EurekaFacts	$ 20,000 $ 168,000	$ 20,000 $ 168,000
Design, prepare for, and conduct cognitive item playtesting sessions (including recruitment, incentive costs, data collection, and summary of findings)	ETS	$ 504,114	$ 674,411
Design, prepare for, and conduct cognitive item cognitive interviews (including recruitment, incentive costs, data collection, analysis, and reporting) Prepare for, and conduct cognitive item cognitive interviews (including recruitment, incentive costs, data collection, analysis, and reporting).	ETS EurekaFacts	$ 380,320 $ 390,701	$ 469,677 $ 485,687
Design, prepare for, and conduct scoring and analysis of cognitive item tryouts. Prepare for and conduct cognitive item tryouts (including recruitment, incentive costs, data collection, reporting).	ETS EurekaFacts	$ 404,593 $ 392,587	$ 508,959 $ 520,109
Total		$ 2,260,315	$ 2,846,843

Table 9. High-Level Schedule of Milestones

Activity	Original Dates	Revised Dates
Cognitive interviews for survey questions	November 2015 – February 2016	November 2015 – February 2016
Pretesting for mathematics and reading cognitive items at grades 4 and 8	November 2015 – February 2016	November 2015 – July 2016
Pretesting for mathematics and reading cognitive items at grade 12	October 2016 – February 2017	April 2016 – April 2017