NAEP 2011 Writing Assessment Audiovisual Stimuli Cognitive Interviews

According to the Paperwork reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 1.5 hours including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].

Protocol Structure

This section is an overview of the protocol structure for both 8^th and 12^th grades. Following this overview is a chart showing each step in the 90-minutes protocols, the estimated time for each stage, and a general description of what will be done. The remainder of this volume consists of the detailed scripts, questions, and probes for each step of the protocol.

All sessions begin with obtaining signed student assent to participate in the research. The interviewer will administer the:

student assent;
introduction to the session;
writing task tutorial; and
tutorial usability questions

After going through the tutorial and answering a brief set of questions about it, a random half of the students (Group 1) will be given a usability test in which they will be asked to use certain features demonstrated in the tutorial. The other half of the students (Group 2) will proceed to the two writing tasks and then take the usability test.

Division into two groups:

Group 1		Group 2
Usability test		Two writing tasks
Two writing tasks		Usability test

Both groups go to the think aloud on the un-timed prompts, debriefing on those prompts, and a final debriefing on the entire session.

Group 1		Group 2
Think alouds on two un-timed prompts		Think alouds on two un-timed prompts
Debriefing after each un-timed prompt		Debriefing after each un-timed prompt
Preferred task		Preferred task
Final debriefing		Final debriefing

The Stages of Protocol Administration

Group 1

	Estimated Time (in minutes)
Student assent	2	Student reads and signs assent form.
Introduction to the Session	3	Interviewer describes the session and answers any questions.
Tutorial Usability Questions	5 5	Student goes through the tutorial & answers set of questions about it.
Tutorial Usability Test	5	Group 1 students are given an AV prompt and asked to think aloud while using 5 specified computer functions.
1st Writing Task	15	Student describes plan for writing. Writes to task.
1st Debriefing	5	Student answers debriefing questions.
2nd Writing Task	15	Student describes writing task. Writes to task.
2nd Debriefing	5	Student answers debriefing questions.
Break	10
3rd Task Prompt	5	Student thinks aloud while reviewing the prompt and planning the writing. Debriefing.
4th Task Prompt	5	Student thinks aloud while reviewing the prompt and planning the writing. Debriefing.
Preferred task	5	Student chooses preferred task and explains his choice.
Final debriefing	5	Final comments

Total 90 minutes

Group 2

	Estimated Time (in minutes)
Student assent	2	Student reads and signs assent form
Introduction to the session	3	Interviewer describes the session and answers any questions.
Tutorial Usability Questions	5 5	Student goes through the tutorial & answers questions about it.
1st Writing Task	15	Student describes plan for writing. Writes to task.
1st Debriefing	5	Student answers debriefing questions.
2nd Writing Task	15	Student describes writing task. Writes to task.
2nd Debriefing	5	Student answers debriefing questions.
Tutorial Usability test	5	Group 2 students are given an AV prompt and asked to think aloud while using 5 specified computer functions.
Break	10
3rd Prompt	5	Student thinks aloud while reviewing the prompt and planning the writing. Debriefing.
4th Prompt	5	Student thinks aloud while reviewing the prompt and planning the writing. Debriefing.
Preferred task	5	Student chooses preferred task and explains his choice.
Final debriefing	5	Final comments.

Total 90 minutes

Cognitive Interview Protocol

1. Student Assent

Interviewer: The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner. Note: Text to be read aloud by the interviewer appears in italics.

Hello, my name is <name >.

Thank you for coming here today to help us.

I know that you’ve been told about what we’re going to be doing in this study. But before we get started, I need you to read this description of the study and ask about anything that is not clear. Then, if you want to continue, you’ll sign the form and we’ll start.

(See Assent form in Volume I Appendix A)

2. Introduction to the Session

I want to tell you a little more about what you’ll be doing.

A new way to test [8th or 12th grade] students’ writing has been developed and we’ve asked some [8th or 12th] graders to try it out. We want to know what you think about the test. You will not be graded on the test. I just want to know what you think about the instructions and writing tasks.

There are no right or wrong things to think. I want you to be as open and candid as possible.

This new test is on the computer. The instructions about how to do the test are also on the computer.

You’ll be going through the instructions first.

After you do this, I will ask you some questions about the different parts of the instructions.

This interview is being recorded so researchers can review the tapes later. Abt Associates, Inc. is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.

Altogether, this will take no more than 90 minutes.

Do you have any questions?

3. Tutorial

Interviewer: All students will go through the tutorial on the Abt desktop computer. In half of the sessions, the student will be asked about the tutorial and then proceed to the first writing task. In the other half of the sessions, the student will be asked about the tutorial and then will be asked to use some of the features that the tutorial demonstrated before proceeding to the first writing task.

Instruction to student:

This is the first time this test has been done on the computer. The test developers thought it would be helpful to demonstrate and explain some computer functions you can use when taking the test.

You’ll see a short demonstration and description about how to use these functions when you take the writing test. This will take just a few minutes.

Let me know when you’ve finished going through it.

4. Tutorial Usability Questions

Questions for All Students

Interviewer: I’d like you to answer a few questions about the tutorial you just went through.

The tutorial described several different computer functions.

1. In general, how useful was the tutorial in learning how to use the writing test functions?

Very useful — go to Q3
Somewhat useful — go to Q3
Not too useful — go to Q2
Not at all useful — go to Q2

2. What would make it more useful?

3. How clear were the different descriptions of the features? Would you say that:

Nearly all of the descriptions were clear
Most of the descriptions were clear
Most of the descriptions were unclear
Nearly all of the descriptions were unclear

4. What about the speed of the tutorial? Does it go:

Much too fast
A little too fast
About right
A little too slow
Much too slow

5. How comfortable do you feel about taking this test now that you’ve watched the tutorial?

Very comfortable — go to Q3
Somewhat comfortable — go to Q3
Not too comfortable — go to Q2
Not at all comfortable — go to Q2

6. How important do you think it is to go through this tutorial before taking a writing test on a computer?

Very important
Somewhat important
Not too important
Not at all important

Tutorial Usability Tasks (Group 2)

We want to let the people who developed this tutorial know how well or poorly the tutorial helps students understand and remember how to use the different computer functions. The developers want to know how the tutorial might be improved.

There are a couple of things I’d like you to do to:

First, you will be shown a writing task. In addition to the written description of the task, there are also some audio-video features.

I’ll ask you to use a particular computer function. It helps us understand how a student might use a function or how to improve it if we know what the person using it is thinking.

As you do each task I ask of you, I’d like you to think aloud. All that means is to just say what you’re thinking about as you use the computer. Please tell me whatever is going through your mind as you do the task.

Think Aloud Practice

Sometimes it’s helpful to do a short “think aloud” practice. I’m going to ask you a simple question and ask you to think aloud as you decide on your answer. The question is:

“How many windows are there in the house or apartment where you live?”

Interviewer: Some students will be silent after hearing the question. Immediately encourage the student to say whatever he/she is thinking. You may need to remind the student that the answer to the question has to be a number.

After you finish a task, or after you’ve gone through all the tasks, I’d like to hear any other comments — good or bad— you have about how the tutorial works, any changes that would make it work better, or more helpful.

Open AV Prompt #________________________________________________

Interviewer: Remind the student to think aloud if he or she is silent for more than a few seconds. Select 5 Tasks from the list below.

Tasks

First, I’d like for you to:

Start the video
Turn on the sound to read the text.
Back up and pick just one sentence for it to read
Change to full screen and back again
Show how much time is left for this task
Open the “writing” panel
Choose a color scheme
Open the Help page
Turn the text captions on/off
Replay the video

5-8. Writing Task Questions

The protocol will be adapted to each specific writing task by the use of generic probes in addition to prompt specific probes. Examples of prompt-specific probes for each grade are presented following the generic protocol.

Interviewer Script

Interviewer: The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.

We want to learn what students think about some writing tasks and how those tasks are explained. One way we do this is to ask each student to “think aloud” starting when the task is presented to them, and as they decide what they will write. All I mean by “think aloud” is to simply tell me what you’re thinking as you see how each task is described, and what you’re thinking you will write.

Think-aloud Practice for Group 1

Sometimes it’s helpful to do a short “think aloud” practice. I’m going to ask you a simple question and ask you to think aloud as you decide on your answer. The question is:

“How many windows are there in the house or apartment where you live?”

Now, you’re going to be given two writing tasks.

But instead of simply reading the task, and then starting to write, I’d like you “think aloud” as you go through the description of each task.

The writing tasks are described in different ways for different tasks. As you saw in the tutorial, some tasks have only a written description. But some tasks have illustrations, or photos, or video clips in addition to the written description.

I am interested in

what you think about how the task is described
and what you plan to write

You will be writing on the computer. You will have 15 minutes to write for each task. You will not be graded on what you write.

There are no right or wrong ways to think aloud. Whatever you are thinking about the task, or about anything you see or hear on the screen as you decide what you will write is all I want to know.

I will also have a few questions after you have finished writing.

Your thoughts about these tasks and how they are presented will help NAEP improve how writing tasks are described so that students can easily understand what they are being asked to write about.

So, do you have any questions before we start?

Generic Probes for Writing Tasks 1 & 2

As the tutorial you just went through said, each writing task will be described and you can use the paper to plan your writing, but you do not have to use it. Remember you can go through the description of the task again if you like. You will have 15 minutes to complete the writing task.

Interviewer: Wait for the student to signal that he or she is prepared to begin writing.

Let me know when you’re ready to start writing.

Before you begin to write— without looking back at the instruction— I’d like you to tell me what the task wants you to write about.

Common Set of Questions

After completing the writing task, the student will be asked a common set of questions (to allow comparisons across types of prompts) and a set of debriefing questions based on the type of prompt. The general template for these questions is given below. For the tasks to be used in the study there will be debriefing questions tailored to the particular prompt. An example for each type of prompt at each grade is given below.

Post-writing Task Debriefing Questions for Two Writing Tasks

Interviewer: If the student answers a question as part of her response to an earlier one, it is not necessary to ask it again. Use non-directive probes and “silent probes” to encourage the student to express all their reactions.

Generic Probes

Generic Text Prompt Probes

Interviewer, note the name and number of the AV prompt questioned in each phase.

I want to get your thoughts about how this writing task was presented.

Thinking about that first screen, what are the main things you remember?
What were your main impressions of what you were supposed to write?
What part(s) of the description were most helpful/least helpful? Why was that?
What information given in the description could be used in the writing task?
Did you think the description made the writing task interesting or not?

Generic AV Prompt Probes

Interviewer, note the name and number of the AV prompt questioned in each phase.

I want to get your thoughts about how this writing task was presented.

What are the main things you remember from what you just went through?

Was there any part of the description that was confusing or just not helpful?
What do you think the purpose of the video is?
Did the video give you ideas about things to include or what to emphasize in your essay?
What information given in the video could you possibly use in the writing task?

Generic A Prompts

Interviewer, note the name and number of the A prompt questioned in each phase.

I want to get your thoughts about how this writing task was presented.

Thinking about the first screen, what are the main things you remember?

Was there any part of the description that was confusing or just not helpful?
How interesting do you think this writing task is?
What do you think the purpose of the audio is?
How did the audio affect your understanding of the writing task?
What ideas did the audio give you about things to include, or what to emphasize in your essay?
Was there information given in the audio that you used in what you wrote?

G Prompts

Interviewer, note the name and number of the G prompt questioned in each phase.

I want to get your thoughts about how this writing task was presented.

Thinking about the first screen, what are the main things you remember?

Were you given a clear description or not of what you were supposed to write?
Was there any part of the description that was confusing or just not helpful?
Did the picture give you ideas about things to include or what to emphasize in your essay?
Why do you think the illustration or photo was included?

Prompt Specific Prompts

In order to better understand how the introduction of graphics, audio, and video affect student responses, questions will be asked based upon the specific prompts for each grade. The generic probes will allow some comparison of in what ways, if any, the additional media affect student performance. The prompt-specific probes will help us understand how particular features of a probe may affect task performance.

Stimulus	Grade 8 Example Prompts	Grade 12 Example Prompts
AV	99887 Prompt 1: What are the main things the calendar illustrated?	94077 Prompt 1: After seeing the images of young people using technology, what did you think about choosing one kind of technology to write about?
-AV	97410 Prompt 5: Please explain this task as though you were explaining it to another student	94077 Prompt 1: What kinds of technology did you think about when deciding what to write?
G	99294 Prompt 2: How do the two newspaper letters compare?	94132 Prompt 6: What do you remember about the picture that was shown?
A	97792 Prompt 6: What do you think the purpose of the audio recording was?	NA
-A	97792 Prompt 6: What kinds of experiences did you consider writing about?	NA

BREAK

9-11. Third and Fourth Prompts

Interviewer:

For these two prompts, the student will plan what he/she would write.

Now I’d like for you to look at two more writing tasks. This time you will not do the writing task. You will plan what you would write. You can use the paper to help you, but you don’t have to use it.

I want you to think aloud again, and tell me what you’re thinking as you go through the instructions and plan what you would write. I’d like you to say everything that comes to mind from the time you start the instruction until you have planned what you would write.

If you had your choice of doing only one of these writing tasks, which one would you choose?

And why is that?

12. Final Debriefing

Before we finish, I’d like to ask just a couple more questions and to hear any other thoughts you have about what you’ve been doing.

Of the four writing tasks--- two that you actually wrote and two that you just did the planning for--- which one do you think was described most clearly? How did you decide on that one?

Of those same four tasks, which one gave you the best information to actually use in your writing?

Thank student for his/her time. Provide gift card for the student and check for the parent and get signature to acknowledge receipt.

File Type	application/msword
File Title	AV prompts:
Author	BlairJ
Last Modified By	Ricardo Martinez
File Modified	2010-05-22
File Created	2010-05-22