National Assessment of Educational Progress (NAEP) Cognitive Items Cognitive Interview Studies 2011

The Paperwork Reduction Act and OMB confidentiality texts are indicated below. Appropriate sections of this information are included in the consent forms and letters contained in the appendices for Volume I. The entire statements will be included on the front of the test materials used in the interviews.

Grade 4 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 60 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

Grade 8 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 1.5 hours including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Cognitive Laboratory Script - Writing

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. Text highlighted in red or placed in brackets is generic text that will be tailored to suit the particular task being studied.

Introduction:

Hello, my name is ______ and I work for <Company Name>. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?
- If student responds with “language arts/reading/writing,” follow up with: Good, then I think you’ll enjoy what we are going to be doing today.
- If student responds with another subject, respond accordingly (e.g., if the student responds with “science,” ask them what they are studying in their science class).

Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study about grade 4 writing tasks, which is part of the National Assessment of Educational Progress, or NAEP for short, overseen by the National Center on Education Statistics, of the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will be doing one writing task.

It’s okay if you do not know how to do the writing task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This new test is on the computer. The instructions about how to do the test are also on the computer.

This interview is being recorded so we can review the tapes later. What you say may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573)..

You’ll be going through the instructions first. After you do this, you will do a writing task.

We will work together for about an hour.

Do you have any questions?

Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue with the writing tutorial. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

OK, let’s begin.

Writing Interface Tutorial

All students will go through the tutorial on the computer to enable them to do the tasks. The script for this part is below.

This is the first time this test has been done for your grade on the computer. There are directions to show you how you to use the computer and some of the tools you have for writing.

You are going to go through the directions. This will take just a few minutes.

Start the tutorial for the student.

At the conclusion of the tutorial:

Do you have any questions before we continue?

Think Aloud Examples – Writing

4a. Think Aloud Modeling Script

You are going to be doing a writing task in a way that may be different from what you are used to. Instead of working quietly, I want you to read the task out loud and then think out loud by telling me what you are thinking as you work on the task. That includes any planning you may do for your writing, and anything you are thinking as you come up with ideas and write.

I will give you an example of the think-aloud process. Then I will give you a chance to practice it.

You will not be graded on anything you say while you are thinking aloud. There are no wrong thoughts, and everything you think and say is important to us.

Now I’m going to show you how to think out loud. When I am finished you will get a chance to practice thinking aloud.

Think Aloud Explanation:

Since we can’t tell what is going in your brain, we need you to “think aloud.”

Let me give you an example about making a piece of toast this morning. If I were thinking aloud while I got the toast ready, it would sound something like this.

“What do I want...ok, I want toast. What kind? Maybe this kind with raisins…I love raisins. OK, first plug in the toaster oven, turn it on, close it. OK, done. Hmm (reading the bread package). How come they always have to advertise using this thing about low fat? Wait…serving size 1 piece of bread, Calories: 130… does not seem so bad…Vitamin A 0%, Vitamin C 0%, Iron 8%. So those are good. (Look up at the corner of the ceiling) Wait! Forgot to put the bread in the toaster! OK, OK, put bread in toaster…ow, hot, hot! Do I want butter or…(Ding) Open the toaster, don’t burn myself…I think I want butter, yeah. I’m really hungry…OK, toast is ready, have butter…this is good.”

The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do. Just say out loud the words that are in your head.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are thinking in your head that you are confused and times when you are not.

4b. Student Practice

You may need to remind the student to talk aloud as he/she works through the question. If necessary, use the “Think-Aloud Hints” shown below to prompt the student, but be careful not to lead the student.

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.

Now you will do a think-aloud practice. I’m going to ask you a question and you think aloud as you figure out your answer. Remember to say out loud whatever you are thinking. The question is:

“How many windows are in the house or apartment where you live?”

Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number.

4c. Additional Interviewer Model

[Use only if you feel the student needs to observe another think aloud before moving on to the actual questions.]

Shape2

[Remember: the italicized text should not be read word for word. You should be familiar enough with its contents to demonstrate a think aloud in a natural and conversational manner.] Hand the student a copy of the item.

Read both the question and the answer choices aloud.
Make sense of the question:

The question is asking me which of the following could be the length of the pencil I use in school.

Analyze each option:

All the options have the number 6 in them, but the units are all different.

Option A says 6 feet, which doesn’t make sense because that would be too big.
Option B says 6 pounds, [refer back to question] but the question asks about the length of the pencil and pounds are about weight.

Option C says 6 ounces, which is also weight.
Option D says 6 inches, [Look at your pencil] which would make sense for the length of a pencil.
My answer is D. [Fill in oval for option D on question.]

When you answer the questions I’d like you to say out loud everything that you’re thinking.

Do you have any questions before we continue? [Answer any questions the student may ask.]

Okay, let’s move on to the actual writing task.

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and do the writing task. Remember, you are not getting a grade on the task you do today.

4d. Think-Aloud Hints for the Interviewer

If the student is silent for 5 or more seconds, use the following as a guideline for encouraging the student to say their thoughts, or to help him/her elaborate their responses.

We’re interested in capturing all the participant’s mental processes while doing the task. Your goal is to have the participant speak aloud all his or her thoughts while doing the task. Grade 4 students may pose special challenges, as it may be hard for them to both work on a task and say their thoughts aloud at the same time. Do your best to encourage them, but be aware that you may learn more during the retrospective interview.

If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouths.” Don’t ask questions that lead the participant’s response or otherwise make them rush or change their approach to writing—such as planning before they write. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

What are you thinking now?
Any other thoughts?
[If there is stimulus]: I see you are looking at the video/pictures/text. What are you thinking?
I see you have stopped writing. What are you thinking?
I see you are reading the brochure. What are you thinking?

NOTE: If the student simply cannot think aloud and write at the same time (judged by their ongoing difficulty doing so, and discomfort with doing do so), let them write, and then rely on the retrospective interview for information about the student’s reaction to the task.

Generic Writing Protocol

Writing Cognitive Lab

NAEP Writing

2011 Grade 4

Question Identifier: Task Number

Note: The preliminary information should be recorded for a task.

Ref #	Question/Prompt	Response
	Preliminary Information
1	Name of Interviewer
2	Date
3	Student ID	Record student’s unique ID number here.
4	Student’s Gender	 Male  Female
5	Special circumstances that may have affected the interview
6	Do you like studying language arts/reading/writing?	 Yes  No

Now, you’re going to do a writing task.

Instead of just reading the task and doing it, I’d like you to “think aloud” as you do the task. All that means is to just read the task out loud and then say what you’re thinking as you do the task. Please tell me whatever is going through your mind as you do the task.

As you saw in the directions on the computer, some tasks have illustrations, or photos, or videos. You may have a task like that.

You will be writing on the computer. You will have 25 minutes to do the task. You will not be graded on what you write.

There is no right or wrong way to think aloud. I want to know whatever you are thinking as you read the task and look at any videos or pictures. I want to know whatever you are thinking as you do the task.

I will also have a few questions after you have finished writing. Your thoughts about these tasks will help NAEP make the tasks better.

So, do you have any questions before we start?

Give the student paper and pencil and the planning and revising brochure.

Ok, first, read these task directions out loud.

[The directions that will be displayed on the screen are shown below.]

You are now ready to complete a writing task. You may use the paper and pencil you have been given to plan your writing, or you may plan on the computer. You may want to use suggestions from the Ideas for Planning and Reviewing Your Writing brochure you have been given.

Be sure to use the panel labeled "Write" to write your response. Some tools, like spell check and thesaurus, are available to you. You may use these tools if you wish, but you do not have to use them.

Click NEXT to continue.

After the student clicks NEXT, continue with the Task and associated questions.

Ref #	Question/Prompt	Response
	PHASE 1: Student reads task aloud
7	Please read the task aloud. OR, If stimulus that can be read is present: Please read this task and all the other words that are presented aloud.	Record your observations when the student reads the question.
8	Without looking back at the task— I’d like you to tell me what the task wants you to do. If there are visual or multimedia stimuli or other components: Without looking back at the task— I’d like you to tell me about the pictures or video or presentation. IF student does not indicate audience in his or previous comments, AND the task indicates an audience, ask: Describe who you are supposed to be writing for or to.
	PHASE 2: Student thinks aloud while doing task
9	Now please go ahead and start work, telling me everything you are thinking. I will be taking notes while you are working and talking.
	If necessary, remind the student to say aloud what his/her thought processes are while working. Use the “Think-Aloud Hints” as needed.	Record student’s think-aloud comments here.
10	Does the student plan? Does the student use the brochure?	 Yes  No  Yes  No
11	How would you rate the student’s level of motivation while working on the task?	 High  Moderate  Low
12	Based on your observations, did the student demonstrate understanding of the task?	 Yes  No  Student demonstrated some understanding.
13	[If task features drag and drop images; if not, move to Phase 3] Based on your observations, did the student demonstrate understanding of using drag and drop images in his or her writing?	 Yes  No  Student demonstrated some understanding.
	PHASE 3: Interview with student
14	If a student in your class didn’t understand the task, how would you explain it? If a stimulus is present, point to the stimulus and ask: How would you explain this image, text, video or presentation to another student?
15	If the task features a presentation, text, or video with information the student can be expected to use in his or her response (versus passive stimulus): What did you learn from this video/presentation/text?
16	Before you started writing, what did you think about? If a stimulus is present, point to the stimulus and ask: Did you think about this picture/video/text/presentation before you started writing? If yes: Describe what you thought.
17	Did you plan what you wrote? If yes: Tell me what you thought about when you planned. If a stimulus is present, point to the stimulus and ask: Did you think about this picture/video/text/presentation when you planned? If yes: Describe what you thought.
18	If task features drag and drop images, point to the images and ask: What did you think you were supposed to do with these pictures? Did you use them in your writing? If yes: Tell me what you thought about when you used them in your writing.
19	Did you change your mind about what you wanted to say after you started to write? If yes: Tell me what you thought about when you did that. If a stimulus is present, point to the stimulus and ask: Did you think about this picture/video/text/presentation when you changed anything? If yes: Describe what you thought.
20	Is there anything that you think could make this task easier to understand? If Yes, What is it? Were there any words you did not understand in the task? If Yes, What were they?	 Yes  No Record student’s suggestions.  Yes  No
21	Have you done tasks like this in school or on other tests? If Yes, Tell me about them. If not already answered and student had drag and drop images, ask: Do you use pictures like this in writing you do in school or at home? If Yes, Tell me about them.	 Yes  No  Yes  No
22	Have you studied writing in school? What kinds of feedback does your teacher give you about your writing? If student does not know word “feedback,” say “comments” from the teacher.	 Yes  No
	Thank you! We are all done.

Behavioral Coding Hints for Interviewer

It is important to note when certain behaviors occur that could be evidence of problems with a task. When the following behaviors occur, note the occurrence and when it occurred:

Misreading or rereading a word or words in the task
Skipping a word or words in the task

Debriefing and Thank You

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Thank participant for his/her time. Provide gift card.

Cognitive Laboratory Script - TEL

Text written in italics is to be read aloud by the interviewer.

Introduction:

Hello, my name is ______ and I work for <Company Name>. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?
- If student responds with “science” or “social studies” follow up with: Good, then I think you’ll enjoy what we are going to be doing today.
- If student responds with another subject, respond accordingly (e.g., if the student responds with “English” ask them what they are currently studying in their English class).

Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study for a new test of technology and engineering literacy, which is part of the National Assessment of Educational Progress, or NAEP for short, overseen by the National Center on Education Statistics, of the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 throughout the United States. You will be answering a small group of test questions for 8th graders.

It’s okay if you do not know how to answer the questions. I will not be grading your work today, and no one will know that it was you who did the questions. I’m not evaluating you. The goal is to see if the test questions are good test questions and I need your help to do this. Please try to do your best.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This interview is being recorded so researchers can review the tapes later. Your information may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

We will work together for about an < hour/hour and a half>.

Do you have any questions?

Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue with the think-aloud training. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Okay, let’s begin.

Think Aloud Examples – TEL

7a. Think Aloud Modeling Script

You are going to be doing questions and tasks in a way that may be different from what you are used to. Instead of working quietly, I want you to read each part of the task out loud and then think out loud by telling me what you are thinking as you read and work on the task.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it.

You will not be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

I’m now going to show you how to think out loud. When I am finished you will get a chance to practice answering a question thinking aloud.

Think Aloud Explanation:

Since we can’t tell what is going in your brain, we need you to “think aloud.”

Let me give you an example about making a piece of toast this morning. If I were thinking aloud while I got the toast ready, it would sound something like this.

“Bread drawer. Muffin or white bread? White bread. Where is the knife? Oh, the sink. Cut. Plug in the toaster oven, turn it on, close it. OK, Medium. Hmm (reading the white bread package). Serving size 1 piece of bread, Calories: 130; Calories from fat 10 – I guess that’s not so bad. Vitamin A 0%, Vitamin C 0%, Calcium 8%, Iron 8% So aren’t there B vitamins in bread? Hmm. (Look up at the corner of the ceiling) Wait! I forgot to put the bread in the toaster! OK, OK, put bread in toaster. Do I want butter or…(Ding) Open the toaster, don’t burn yourself! Use the knife to pull out the bread. Table, sit down, eat.”

The point of the think-aloud is not to describe what you are doing or to tell me what you plan to do. Just say aloud the words that are in your head.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I need to hear times when you are confused (and what is confusing you) and times when you understand what you are doing.

7b. Student Practice

You may need to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, use the “Think-Aloud Hints” shown below to prompt the student but be careful not to lead the student.

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.

Now you will do a think-aloud practice. I’m going to ask you a simple question and you think aloud as you figure out your answer. Remember to say out loud whatever you are thinking. The question is:

“How many windows are in the house or apartment where you live?”

7c. Additional Interviewer Model

[Use only if you feel the student needs to observe another think aloud before moving on to the actual questions.]

Shape3

1. Read both the question and the answer choices aloud.

2. Make sense of the question:

The question is asking me which of the following could be the length of the pencil I use in school.

3. Analyze each option:

All the options have the number 6 in them, but the units are all different.

Option A says 6 feet, which doesn’t make sense because that would be too big.
Option B says 6 pounds, [refer back to question] but the question asks about the length of the pencil and pounds are a unit of weight.

Option C says 6 ounces, which is also a unit of weight.
Option D says 6 inches, [Look at your pencil] which would make sense for the length of a pencil.
My answer is D. [Fill in oval for option D on question.]

When you answer the questions I’d like you to say aloud everything that you’re thinking.

Do you have any questions before we continue? [Answer any questions the student may ask.]

Okay, let’s move on to the actual test questions.

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions. Remember, you are not receiving a grade on the questions and tasks you do today.

7d. Think-Aloud Hints for the Interviewer

If the student is silent for 5 or more seconds, use the following as a guideline for encouraging the student to say their thoughts, or to help him/her elaborate their responses.

We’re interested in capturing all the participant’s mental processes while answering questions or doing tasks. Your goal is to have the participant speak aloud all his or her thoughts while doing the questions and tasks. If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouths.” Don’t ask questions that lead the participant’s response or otherwise make them rush or change their approach. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

What are you thinking now?
Any other thoughts?
I see you are looking at the picture/figure/chart/text, etc... What are you thinking?

Also, if it seems a student is hung up on something, it’s important to note their frustration and when it occurred.

Generic TEL Protocol

Cognitive Labs

NAEP TEL

2011 Grade 8

Question Identifier: Question/Task Number

Note: The preliminary information should be recorded for each question/task in this set. Ask student questions 1-8 only once. Scenario-based assessment tasks consist of a series of embedded assessment items; questions 9-22 are to be repeated for each embedded item or other subsection of the task.

Ref #	Question/Prompt	Response
	Preliminary Information
1	Name of interviewer
2	Date
3	Student ID	Record student’s unique ID number here.
4	Student’s Gender	 Male  Female
5	Name of School
6	School District
7	Special circumstances that may have affected the interview
8	Do you like studying subject?	 Yes  No

Now, you’re going to be given [1, 2, 3] task(s) to work on.

Instead of just reading the task and doing it, I’d like you to “think aloud” as you do the task.

[modify as appropriate for the particular computer prototype/mockup:] When we are finished making the test, students will take it on computers. However, we aren’t finished making the test yet, so some parts of the tasks don’t work yet. When we come to one of those parts, you will have to write down your answers on a sheet of paper, or we may ask you to write down your answers using a separate program (e.g. a spreadsheet, word processor, or other software meant to provide functionality that will be built into the final versions of items).

You will have [10, 20, 30, 40] minutes to do this task. You will not be graded on what you do during the task.

There is no right or wrong way to think aloud. I want to know whatever you are thinking as you read each part of the task.

I will also have a few questions after you have finished working on the task. Your thoughts will help us make the tasks better.