NAEP 2011 Writing Assessment Audiovisual Stimuli Cognitive Interviews

National Assessment of Educational Progress

Volume I

Appendixes

Cognitive Interview Study of the 2011 Writing Assessment

Audiovisual Stimuli: Research Design and Protocol

May 19, 2010

OMB# 1850-0803 v.27

Cognitive Interview Study of the 2011 Writing Assessment

Audiovisual Stimuli: Research Design and Protocol

Appendixes

Appendix A Student Assent Form

Appendix B Legal Guardian Consent

Appendix C Interviewer and Researcher Confidentiality Agreement

Appendix D E-mail for Recruiting Participants

Appendix E Confirmation Letter to Participants

Appendix F Confirmation Letter to Parents

Appendix G Confirmation Phone Script

Appendix H Thank You Letter to Participants

Appendix I Thank You Letter to Parents/Participants

Appendix J Student Screening Phone Script

Appendix K Parent Screening Phone Script

Appendix A Student Assent Form

Dear Student: <DATE>

A representative from Abt Associates, Inc. is going to meet with you today to ask you to answer some questions about a new writing test that will become part of the National Assessment of Educational Progress (NAEP) and be eventually administered to students in your grade. This will take about 90 minutes.

The National Center for Education Statistics (NCES), part of the U.S. Department of Education, needs to know if the test instructions and questions are clear. Your ideas will help NCES improve the test so students like you understand all the questions and the things you are being asked to do on the test. You will be given questions that are very similar to the actual questions that will be asked during the national testing. The representative will pay very close attention to your opinions and feelings about all aspects of the test. He or she will also ask what you think about the questions after you complete them.

You will not be judged on how well you perform and you will not receive a grade. Other students like you will also participate in one-on-one sessions just like these. These sessions will all be videotaped for the researchers on this study to review. Abt Associates, Inc. is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].

These interviews are an important stage of the NAEP test development process. Your comments, along with the comments of other students, will help NCES improve the writing assessment. We thank you for your voluntary participation and ask you to please sign the form below.

Thank you,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

I hereby accept the terms described above and agree to participate voluntarily in the interview.

­­­­­­­­­­­­­­­­­­­­­­­­­­

__________________________________

Print Name

____________________________________ ________________________

Signature Date

Appendix B Legal Guardian Consent

Dear Parent or Legal Guardian: <DATE>

A representative from Abt Associates, Inc., on behalf of the National Center for Educational Statistics (NCES), part of the U.S. Department of Education, is conducting interviews with <GRADE> grade students about a new writing test, that will become part of the National Assessment of Educational Progress (NAEP). The interviews will last about 90 minutes. The purpose of these interviews is not to test or grade students. We want to listen to their thoughts about the test instructions and the test questions, and any ideas they have for improving how the test is administered or how the questions can be made easier to understand.

The representative will meet one-on-one with each student to ask for his or her input. These sessions will be videotaped. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your child’s participation is voluntary. Your child’s responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].

These interviews are an important stage of the NAEP test development process. The comments students provide help NCES improve the test instructions and questions to make the best test possible. Your child will be given $30 as thanks for their time.

We hope that you will give your consent for your son or daughter to participate in the interview by signing this form. Without your consent, your child will not be able to participate in the interview.

Respectfully,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

I hereby give my permission for _____________________________________________ to

(Student’s Name)

participate in the NAEP writing assessment interview.

­­­­­­­­­­­­­­____________________________________ ________________________

Signature Date

Appendix C Interviewer and Researcher Confidentiality Agreement

Agreement: Item and Test Security Requirements for the National Assessment of Educational Progress (NAEP) Writing Assessment

Under this agreement, you will be reviewing, creating, or handling secure test material belonging the National Center for Educational Statistics (NCES) in the United States Department of Education. These test materials are confidential and may not be discussed with any person who has not signed this agreement. Any materials associated with the testing of the NAEP Writing Assessment may not be copied, published, announced, or in any other way be made public, and you may not grant permission for anyone else to share any information. If you are creating secure test material under this agreement, for purposes of copyright, it is understood and agreed that any material produced will be considered a work made for hire. As such, the U.S. Department of Education or its designee will be the sole and exclusive owner of the material you produce and will copyright thereon in the United States and throughout the world and you will retain no rights in such materials.

By signing this agreement, you acknowledge that the test materials that are the subject of this agreement are proprietary and confidential materials of the U.S. Department of Education. You further understand that any disclosure, unauthorized use, or reproduction of the materials would damage the confidentiality of NAEP. By entering into this agreement, you agree to comply with the following:

1. Confidential test materials must be only used for the sole purpose of performing professional services for NCES. Any other use of these materials would require prior written consent of NCES.

2. Confidential materials must be kept in a locked container when they are not being used.

3. Confidential materials must not be shared with any parties who have not signed this agreement.

4. Confidential test materials must not be copied in part or whole, and all copies of materials must be returned promptly to the U.S. Department of Education or its designee upon task completion.

5. In the event that confidential test materials are lost, stolen, or an unauthorized party has gained access to these materials, the U.S. Department of Education must be notified immediately.

ACCEPTED AND AGREED TO THE REQUIREMENTS

NAME:___________________________________________________________

SIGNATURE:______________________________________________________

DATE:____________________________________________________________

Appendix D Email for Recruiting Participants

Email subject line: Dept. of Ed. needs your input

Dear <Student Name>;

The National Center for Educational Statistics (NCES), part of the U.S. Department of Education, has developed a new writing test for students that will become part of the National Assessment of Educational Progress (NAEP). Before giving the new test to all students, NCES needs to know if the test instructions and questions are clear. Your input will help NCES improve the test so students like you understand all the questions and the things you are being asked to do.

The interviews will take place in downtown Bethesda and will last approximately 90 minutes. You will be asked about the instructions and to complete a few tasks. You will not receive a grade on any of the questions. NCES simply wants to know how you would answer them if it was an actual test. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183]. To thank you for your time, you will receive a $30 gift card and your parents will receive a $25 check.

These interviews are an important stage of the NAEP test development process. Your comments, along with the comments of other students, will help NCES improve the writing assessment.

If you are interested in participating in this research project, please call [phone number] or email [email address].

Thank you,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

Appendix E Confirmation Letter to Participants

<DATE>

Dear <Student Name>;

Thank you for agreeing to help the National Center for Educational Statistics (NCES), part of the U.S. Department of Education, to refine a new writing test for students, which will become part of the National Assessment of Educational Progress (NAEP). Before giving the new test to all students, NCES needs to know if the test instructions and questions are clear. Your input will help NCES improve the test so students like you understand all the questions and the things you are being asked to do. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].

You have been scheduled for an interview at <time> on <date>. The interview will last approximately 90 minutes. You will be asked about the instructions and to complete a few tasks. To thank you for your time, you will receive a $30 gift card and your parents will receive a $25 check.

The interviews will take place at:

4550 Montgomery Avenue
Suite 800 North
Bethesda, MD 20814-3343.

It is metro accessible and there are also parking garages across the street.

These interviews are an important stage of the NAEP test development process. Your comments, along with the comments of other students, will help NCES improve the writing assessment.

Thank you for volunteering in this very important research. Our interviewer is looking forward to meeting with you. If you have any questions, please contact [contact information].

Thank you,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

Appendix F Confirmation Letter to Parents

<DATE>

Dear <Parent Name>;

Thank you for agreeing for your child to help the National Center for Educational Statistics (NCES), part of the U.S. Department of Education, refine a new writing test for students, that will become part of the National Assessment of Educational Progress (NAEP). Before giving the new test to all students, NCES needs to know if the test instructions and questions are clear. Your child’s input will help NCES improve the test so all students understand the directions and questions. Your child’s answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].

Your child has been scheduled for an interview at <time> on <date>. The interview will last approximately 90 minutes. Your child will receive a $30 gift card as a token of appreciation for their time. In addition, you will receive a $25 check.

The interviews will take place at:

4550 Montgomery Avenue
Suite 800 North
Bethesda, MD 20814-3343.

It is metro accessible and there are also parking garages across the street.

These interviews are an important stage of the NAEP test development process. Your child’s comments, along with the comments of other students, will help NCES improve the writing assessment.

Thank you for volunteering in this very important research. Our interviewer is looking forward to meeting with you. If you have any questions, please contact [contact information].

Thank you,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

Appendix G Confirmation Phone Script

Good morning/afternoon. May I please speak to < student name> about the NAEP Writing Test interview?

Good morning/afternoon. My name is < name> from UserWorks calling on behalf of the National Center for Educational Statistics or NCES, part of the U.S. Department of Education. We are so pleased that you agreed to help NCES in developing the best possible writing test possible. We have scheduled you for <time> on <date>.

The interview will last approximately 90 minutes. It will take place in downtown Bethesda at 4550 Montgomery Avenue, Suite 800 North.

We will be asking you about the test instructions and some of the test questions. NCES needs to make sure that students like you will understand what the test is asking you to do. Your input will help NCES improve the writing test that will be administered nationwide. To thank you for your time, you will receive a $30 gift card. In addition, your parents will receive a $25 check.

Do you have any questions at this time?

[Answer any questions.]

We plan to see you at <time> on <date>. If you do have any questions before then, please call <phone number>.

Thank you again for agreeing to be part of this very important research project.

Appendix H Thank You Letter to Participants

<DATE>

Dear <Student Name>;

On behalf of National Center for Educational Statistics (NCES), part of the U.S. Department of Education, I would like to thank you for your participation in a research study that will help us develop the best possible writing test. Your assistance helped us better understand how students approach both the testing directions and the test questions. Your input, combined with input from other students like you, is incredibly helpful.

Our interviewers found the interviews very useful and informative. Your open, candid and insightful comments were very informative.

Again, thank you for your assistance.

Sincerely,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

Appendix I Thank You Letter to Parents/Participants

<DATE>

Dear <Parent Name>;

On behalf of the National Center for Educational Statistics (NCES), part of the U.S. Department of Education, thank you for allowing your child to participate in writing test research. Your child’s input will help us develop the best possible writing test. The assistance provided by your child helped us to better understand how students approach both the testing directions and the test questions. Your child’s input, combined with input from other students, is incredibly helpful.

Our interviewers found the interviews very useful and informative.

Again, thank you for your assistance.

Sincerely,

Johnny Blair

Principal Scientist

Abt Associates, Inc.

Appendix J Student Screening Phone Script

Good morning/afternoon. May I please speak to <name of student> about the National Assessment of Educational Progress or NAEP Writing Test interview?

Good morning/afternoon. My name is <name> from UserWorks calling on behalf of the National Center for Educational Statistics or NCES, part of the U.S. Department of Education. We have spoken with your <mother, father> and we are pleased that you and <she, he> have agreed that you can help in developing the NAEP writing test.

Before we schedule your session, I need to confirm some information we have and tell you more about what you’ll be asked to do.

1. What grade are you currently in?

2.    Are you of either Hispanic or Latino origin?

Are you:

- American Indian or Alaska Native

- Asian

- Black or African American

- Native Hawaiian or Other Pacific Islander, or

- White?

(You may select one or more)

3. And are you male or female?

In the session, we will be asking you about the writing test directions and some of the test questions. NCES needs to make sure that students like you will understand what the test is asking you to do. Your input will help NCES improve the writing test that will be administered nationwide. To thank you for your time, you will receive a $30 gift card. In addition, your parent will receive a $25 check.

The interview will last approximately 90 minutes and will be recorded. It will take place in downtown Bethesda at Abt Associates Inc., located at 4550 Montgomery Avenue, Suite 800 North. Your participation is entirely voluntary and your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.

Do you have any questions?

[Answer any questions.]

Would you like to participate?

[If No] Thank you for taking the time to speak with us today about this important project.

[If Yes] We would like to schedule your session for < date, time>. Is that day and time ok?

We plan to see you at <time> on <date>. If you do have any questions before then, please call <phone number>.

Thank you again for agreeing to be part of this very important research project.

Appendix K Parent Screening Phone Script

Good morning/afternoon. May I please speak to <name of parent> about the National Assessment of Educational Progress or NAEP Writing Test interview?

Good morning/afternoon. My name is <name> from UserWorks calling on behalf of the National Center for Educational Statistics or NCES, part of the U.S. Department of Education. Thank you for contacting us about the possibility of < student name > participating in a project to help develop the NAEP writing test.

First, I need to confirm the information from your <email, telephone call> indicating your interest as well as that of < student name >. Then I’ll tell you more about what <she, he> will be asked to do.

1. What grade is < student name > currently in?

2. Is < student name > either Hispanic or Latino origin?

Is < student name >:

- American Indian or Alaska Native

- Asian

- Black or African American

- Native Hawaiian or Other Pacific Islander, or

- White?

(You may select one or more)

3. And has < student name > said <she’d, he’d> be interested in participating in the interview session which will take 90 minutes?

In the session, we will be asking < student name > about the writing test directions and some of the test questions. NCES needs to make sure that students will understand what the test is asking them to do.

Participation is entirely voluntary and your child’s answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law. Your child’s input will help NCES improve the writing test that will be asked nationwide. To thank <her, him>, < student name > will receive a $30 gift card. In addition, you will receive a $25 check.

The interview will last approximately 90 minutes and will be recorded. It will take place in downtown Bethesda at Abt Associates Inc., located at 4550 Montgomery Avenue, Suite 800 North.

Do you have any questions?

[Answer any questions.]

Do you think < student name > would like to participate?

[If No] Thank you for taking the time to speak with us today about this important project.

[If Yes] And would you be willing to sign a form indicating your approval for <student name> to participate?

[If No] Thank you for taking the time to speak with us today about this important project.

[If Yes]

At the interview session, < student name > will also be asked to sign a form indicating willingness to participate. In the next day or so, we will call <her, him> to be sure < student name > understands what we’re asking <her, him> to do, confirm that < student name > is still interested, and schedule a time for the session.

If you or < student name> have any questions before then, please call <phone number>.

Thank you again for agreeing to be part of this very important research project.