Passback Q and A document

MEMORANDUM OMB # 1850-0803 v.58

DATE: November 15, 2011

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patricia Etienne

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Response to OMB passback on NAEP Cog Lab Study for NVS Reading Accessible Booklet Study

1. We would not support a plan to take students (especially low-performing ones) out of school day instructional time. We understand the desirability of using a school setting, and are fine with after-school. We hope this is what NCES is planning and would appreciate clarification in the document.

Conducting the cog labs after school would require the parents and students to make other arrangements for transportation. Therefore, we plan to conduct the cog labs during the regular school day, but outside academic instructional times (e.g., during study hall). The scheduling for the cog labs will be coordinated through the school liaisons to ensure that there is no conflict with instructional time. We have added clarifying text to this effect on page 4 of Volume I, stating: “The cog labs will take place outside academic instructional times.”

2. We’d recommend some improvement to address the length and general fed-speak of the intro script and recruitment/consent letters. For the length issue, we suggest reducing the whole sponsorship statement (in the script and all the letters) to just the National Center for Education Statistics and get rid of the “also called NAEP” statement from the script. Also, we have evidence from Census Bureau research that the phrase “statistical purposes” is not well understood by adults (e.g., Gerber 2000, Landreth 2001), who found that alternatives such as “used to produce statistics” were clearer and more reassuring. This seems especially likely to be an issue for children, so would remind NCES that we have an interest in seeing its confidentiality pledge language tested as an add on in a cog lab or two especially for children. Why not start with this lab?

We have revised the script and recruitment/consent letters to reduce the sponsorship statement and limit parenthetical references to NAEP.

We will be happy to include the confidentiality pledge language in a NAEP cog lab. However, we have concerns about exceeding the allotted timeframes for the proposed cog lab activities included in this study. Therefore, we will include cog lab questions on the confidentiality pledge language in our next cog lab submittal.

3. We do not seem to have the actual items being tested, which makes it a little hard to understand the package. If you do not wish for the items to be in the public domain, that’s fine, but please email them to us.

The specific NAEP reading blocks and items are in development/selection process. To facilitate the review and comprehension of the package, we have included two released reading passages and items, one from 4^th grade (Gr.4 Released Reading Block.pdf) and one from 8^th grade (Gr.8 Released Reading Block.pdf), which are representative of those that will be used in the study.

4. It seems like the behavior coding is an afterthought (you provide a note to interviewers after the first item block that gives tips). It seems like making the behavior coding an explicit part of the protocol would be useful given that the overall point of this research is to develop more valid test items that show response to lower levels of reading comprehension. This jumped out for us especially because of the statement that these labs will recruit research associates rather than experienced qualitative researchers, who may not know or remember to take note of the salient behaviors.

As recommended, we have moved the behavior coding section so that it directly precedes the protocol, after the think-aloud hints for the interviewer. Please see page 6 of Volume II.