NAEP Updated Memo

MEMORANDUM OMB # 1850-0803 v. 170

DATE: September 9, 2016

TO: Kashka Kubzdela

National Center for Education Statistics

FROM: Patricia Etienne

National Center for Education Statistics

SUBJECT: National Assessment of Educational Progress (NAEP) Survey Assessments Innovations Lab (SAIL) Pretesting Activities: Virtual World for English Language Arts Assessment – Updated [Revision to a previously approved package (1850-0803 v.106)]

The SAIL Pretesting Activities: Virtual Worlds for English Language Arts Assessment was approved on July 17, 2014 (OMB #1850-0803, v. 106). After completing the playtesting phase of pretesting it has been determined that extended time is needed for the cognitive interview sessions to allow enough time for the think aloud process to take place.

Originally the cognitive interviews were intended to be completed during a 90 minute session. The revised plan increases the cognitive interview session time to 120 minutes. The additional 30 minutes of testing time with student cognitive interview participants is being requested to allow additional time to implement the think-aloud methodology during the cognitive interview sessions (i.e., time for training students on the method and additional time to complete the task while intermittently thinking aloud). In playtesting sessions (without the think aloud method), students have spent on average 75-80 minutes just completing the task, which led us to estimate that more time will be needed for the cognitive interviews. Given that we are increasing the estimated time to complete a cognitive interview by over 33%, we are also increasing the per student incentive for cognitive interviews from $25 to $30 to attract students to participate in a 2 hour session outside of school.

Finally, we are also adjusting the overall schedule for all activities approved for this study to accommodate unanticipated delays in the development of the ELA Virtual World tasks.

Volumes I and II, and Appendices A-F have been revised to reflect these changes. Following are the original and revised burden and cost tables, as well as the revised schedule.

Original Estimates of Hourly Burden for SAIL Pretesting Activities: Virtual Worlds ELA

Estimate of Hourly Burden for Cognitive Interviews

Respondent	Hours per respondent	Number of respondents	Total burden hours
Student Recruitment via School Administrators and Staff and Club and or Community Center Administrators
Initial contact with staff: e-mail, flyer distribution, and planning	0.33	9	3
Parent or Legal Guardian
Flyer review	0.08	180	15
Completion of online screening form or phone screening	0.15	30*	5
Consent form completion and return	0.13	15**	2
Confirmation to parent via email or letter	0.05	30*	2
Recruitment Totals		189	27
Student
Virtual World for ELA	1.5	15	23
Interview Totals		15	23
Total Burden	279 responses	204	50

* Subset of initial contact group ** Subset of students with completed screening forms

Combined Burden for SAIL Virtual World Research Activities

	Number of respondents	Number of responses	Burden Hours
Pretest Activity
Total Play Testing	340	465	81
Total Cognitive Interviews	204	279	50
Total Tryouts	421	423	359
Overall Totals	965	1,167	490

Revised Estimates of Hourly Burden for SAIL Pretesting Activities: Virtual Worlds ELA

Estimate of Hourly Burden for Cognitive Interviews

Respondent	Hours per respondent	Number of respondents	Total burden hours
Student Recruitment via School Administrators and Staff and Club and or Community Center Administrators
Initial contact with staff: e-mail, flyer distribution, and planning	0.33	9	3
Parent or Legal Guardian
Flyer review	0.08	180	15
Completion of online screening form or phone screening	0.15	30*	5
Consent form completion and return	0.13	15**	2
Confirmation to parent via email or letter	0.05	30*	2
Recruitment Totals		189	27
Student
Virtual World for ELA	2	15	30
Interview Totals		15	30
Total Burden	279 responses	204	57

* Subset of initial contact group ** Subset of students with completed screening forms

Combined Burden for SAIL Virtual World Research Activities

	Number of respondents	Number of responses	Burden Hours
Pretest Activity
Total Play Testing	340	465	81
Total Cognitive Interviews	204	279	57
Total Tryouts	421	423	359
Overall Totals	965	1,167	497

Costs to Federal Government:

Original Cost: $489,529

Revised Cost: $489,604

Revised Schedule Milestones for SAIL Pretesting Activities: Virtual Worlds ELA

High-Level Schedule of Milestones

Activity	Dates
Play-Testing	Nov 2014 – August 2016
Cognitive Labs	October 2016 – December 2016
Tryouts	December 2016 – May 2017