National Assessment of Educational Progress (NAEP) Social Studies Pretesting

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average xx¹ minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this information collection, please write to: National Assessment of Educational Progress, National Center for Education Statistics, 1990 K Street, N.W., Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Play Testing Protocol (Social studies)

Introduction to Study and Setup of Item and Task Review Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word-for-word, but should be familiar enough with its contents to conduct the play testing sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

My name is __ and I am from Educational Testing Service. I am here with my colleagues [introduce colleagues]. We are developing a new set of tests in social studies for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. We are going to ask for your opinions about some interactive questions and tasks that we are creating for a NAEP [civics AND/OR geography AND/OR U.S. history] test. Right now, we have some early drafts of what these questions and tasks might look like. You are here to give us your thoughts about these questions and tasks. We’d like to hear about what you think is interesting, what you think is not interesting, and if you think any part of a question or task is confusing.

My colleagues and I will take notes on what you are saying about the questions and tasks—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts about these questions and tasks. There are no right or wrong thoughts.

What you say will be used only to improve our online assessments. What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. We will look at what you say later, but only so that we can understand how our test is working and how we can make it better. Overall, this session should take about [indicate correct length] minutes.

As you work through these questions/tasks, we want you to talk with each other as you go along. We might also ask you a few questions now and then.

The facilitators will generally observe the students with little interruption and will listen to the conversation that occurs between group members for insights into what the students are thinking about the questions and tasks. Intermittently, as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this question/task interesting? Why? /Why not?
Are there any questions or words that seem confusing here? Did you understand that part?
How would you answer this question? [Ask different group members if their approaches would differ].
How could this question/task be improved? Could it be clearer, or more interesting, for example?
Were there questions in the task you did not like? Why?
This question/task asks you about <insert focus of question or task>. Have you learned about this before, either in school or in your own life?
Did you like working with the tablet? Why?
Did you find the directions for how to do the task helpful or not? Why?

Facilitators will observe students, take notes, and pay attention to verbal answers to the questions above (and write them down), and they will look for evidence of engagement, boredom, or confusion in what students say to each other and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing play testing sessions will introduce a question to groups of students talking about the items or tasks to get more information from students, particularly in cases when students talk about issues related to their interest (or non-interest) in the item or task, confusing language in the item/task, or how they might respond to a question on the task. In such cases, observers might ask something like, “Can you tell me more about what you said?” Prior to each play testing session, ETS staff may informally identify some key focus areas for each item/task. If students do not provide sufficient comments on targeted parts, a staff member may ask a group of students if they had any thoughts about the particular sections, using questions such as those described above but focused on specific places or issues in the item or task.

At the end, students will be thanked for their participation and for helping us to improve our tests. Gift cards will be provided at the conclusion of each play testing session.

Part C. Cognitive Interviews (social studies)

I. Introduction to Study Script

NOTE: The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied. In general, protocols will be tweaked and customized in light of specific task content and greater knowledge of delivery devices.

Introduction:

Hello, my name is ______ and I work for [indicate ETS or EurekaFacts]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out what students know in [civics AND/OR geography AND/OR U.S. history]. It is part of the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about [indicate correct length] minutes.

It’s okay if you don’t know how to do any parts of the task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the tablet. The instructions about how to do the test are also on the tablet.

This interview is being recorded so we can review it later. We will also be able to capture what you tap on and where you move your fingers. What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together on [specify number of items/tasks].

Do you have any questions?

After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his/her time and end the interview.

Okay, let’s move on. Before we look at the real tasks, I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real tasks. They should help you understand how we want you to respond. Do you have any other questions before we start?

NOTE: Check the study schedule to establish whether the student should undergo the assigned tasks with a concurrent or retrospective verbal protocol (think-aloud) procedure. If concurrent, the student will think aloud while he or she is doing the task; in those cases, follow the steps described in section II. If retrospective, the student will describe his or her thinking after he or she has completed the task, while watching a recording of the task that was made as they proceeded through it; in those cases, follow the steps described in section III.

II. Concurrent Think-Aloud

II a. Concurrent Think-Aloud: Instructions and Modeling Script

NOTE: If the schedule indicates that the retrospective method is to be used, please skip this section and move to section III.

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular task being studied.

To help us make our test better, we will ask you to complete some tasks. While you are doing the tasks, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the tablet such as what you tap on and anything you put into text boxes.

I’m going to ask you to do the questions in these tasks in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the task. We call this “a think aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you are thinking as you are working on the task. When I am finished I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to think aloud.

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task, and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar.

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so I am reading the question…. It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering what do they mean by “similar”. That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So the choices are beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… oh right, but dolphins are mammals too, aren’t they? Yes, I think they are… but really they seem so different from mice and cows to me. I don’t know… this seems like a hard question… I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for… mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not really similar! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and that kind of thing… so, yes, even though it was really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So those things seem like they are important, and that is how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the tasks today. The point of the think-aloud is to get at whatever is in your head as you are doing the tasks. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the tasks.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he or she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, interviewers should use the “think-aloud hints” given below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud to tell me all of the thoughts that are in your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions, say: Good, then let’s begin our study.] [If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address the concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him or her to stop.]

II c. Concurrent Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa].)

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—tell me what you are thinking as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels OK and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him or her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he or she does not wish to continue or does not feel comfortable continuing, allow the student to stop.]

II d. Concurrent Think-Aloud: Starting the Tasks

Now we will move on to the actual task. Remember, as you do all the activities and answer all the questions in the task, I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet. This task should take about [indicate correct length] minutes. Remember, you will not be graded on what you do during the task and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the tasks better. I will also have a few questions after you have finished working on the task.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Here is the task. When you are ready, go ahead and start working on it.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of think-alouds is to capture all the student’s mental processes while working through tasks. Interviewers must strive to have the student speak aloud all of his or her thoughts while doing the tasks. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate on his or her responses.

If the student is not verbalizing enough, interviewers should offer a verbal nudge to remind the students to keep talking, such as:

What’s going on in your head right now?
I see you’re looking at the task [or screen/figure/chart/text]. What are you thinking?
You seem to be thinking hard—can you tell me what’s in your mind right now?
What are you thinking now?
Any other thoughts?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot tell any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed and pushing them should be avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to state what is in their mind beyond a basic reply such as “I am reading the question.” Interviewers must be sensitive to nonverbal signals that may indicate students cannot say any more than they have or that they need a few extra seconds to gather their thoughts to put them into words.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors during Tasks

As a student is proceeding through a task, a screen-capture system will record evidence of his or her actions on the task (such as where a student taps the screen, how he or she moves through the content and the questions, and any text or numbers entered into fields or boxes). During the session the interviewer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of the student’s understanding, engagement, or use of the task. The following are examples of such behaviors for interviewers to note.

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided in the task by tapping on tabs or buttons to reveal additional or earlier information, digital notepads, digital calculators, or reminders of what to include or think about?
Does the student rapidly move through the screen or take his or her time? Does the student spend a lot of time on a particular aspect?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section IV).

III. Retrospective Think-Aloud

III a. Retrospective Think-Aloud: Instructions and Modeling Script

Using the retrospective (versus concurrent) think-aloud approach, students are instructed to complete a task as they would under normal testing conditions (silently, without interruption). Then, after the task is completed, they will be asked to describe what they were thinking while reviewing the task on the computer. A replay of the task (e.g., using software like Camtasia^® or Morae^®)^² will help cue the student’s recall of the thought processes that occurred during the task as the student progressed through it.

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, interviewers should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that can be tailored to suit the particular writing task being studied.

To help us make our test better, we will ask you to complete some tasks. While you are doing the tasks, we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the tablet, such as what you tap on Overall, this session should take about [indicate correct length] minutes.

First you’ll do the task, working at your own pace through all of the activities and questions you will see on the screen. Then, after you have finished, I’m going to ask you to go back over the task with me. As we go back over it and look at the task again, I will ask you to tell me what you were thinking as you were working through each of the sections. We call this thinking aloud, because we are asking you to say everything you were thinking during the task out loud. To help you remember what you were thinking as you worked on it the first time, we will look at the recording we just made of you doing the task. This way, you will be able to see all of your screen actions and all of the answers that you gave as you were doing the task. We will watch the recording, and as we are going through it, I will be asking you to tell me what you were thinking at different points. Does that make sense? Do you have any questions?

[Answer any questions and clarify the sequence of steps, if needed.]

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what you were thinking while you were working on the task. When I am finished, I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to think aloud.

Let me give you an example. [Place example question in front of student.] Look at this question. It asks me to choose which two animals are the most similar. I’m going to do this question just as I would for a test. After I have finished, I will try to remember what I was thinking as I was working on the question so I can tell you.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar.

Beetle
Mouse
Crab
Dolphin
Cow

[The interviewer ‘works’ silently for about a minute on the question. The interviewer should enact some behaviors that suggest that he or she is thinking hard about the question and carefully considering the possible answers; for example, pointing with a pencil tip at each option in turn; moving a pencil or fingertip between answer options: returning to the question; giving nonverbal signals indicating consideration, uncertainty, etc. Finally, the interviewer circles two answers, B and E, and then begins the retrospective think-aloud.]

Okay, so here I was reading the question…. It says I have to decide on the two animals that are the most similar. So at first I was wondering what do they mean by “similar.” That could mean anything. And I was thinking I wasn't sure how I’m going to make that decision. So... the choices... beetle, mouse, crab, dolphin, and cow. Yeah, well, as I'm reading the choices I'm already thinking there are some things about them that are similar, but there are also differences. I was thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. And the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Oh yeah, and then there is the dolphin that lives in the water, too… and I wondered about that, but that’s about all it has in common with the crab, so I couldn’t say they're very similar even though they’re both in the sea. So in the end I thought the mouse and the cow are both mammals, they both have fur, and they have live babies instead of laying eggs or whatever… but then I thought, dolphins are mammals too, right? I think they are… anyway they seem so different from mice and cows to me. So I wasn't sure… it seemed like a hard question… I wasn't sure what to choose, but in the end I had to make a decision, so I decided to go for mouse and cow. Because they both have fur and four legs and walk around on land. It’s weird because they’re really not that similar. But, when I think of all the others, they seem really different from each other like in where they live and their bodies. So even though it was really hard to decide, that seemed like the best choice to me, because they both live on land and they’re both mammals. So those things seemed pretty important and that’s how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about your social studies task today. The point of the think-aloud is to get at whatever was in your head as you were doing the task. Just say aloud the words and the thoughts that were in your head, as you were thinking and working.

OK, let’s have you give it a try.

III b. Retrospective Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he or she can read it. Allow the student as much time as needed to answer the question. When he or she indicates that he or she has finished, ask the student to begin thinking aloud. Some students will be silent after reading the question. Such students should be asked to say whatever they are thinking. It may be necessary to remind students to talk aloud as they work through the questions and tasks. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt students, being careful not to lead students. The interviewer needs to be familiar enough with the information to encourage the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. I will give you a question like the one I just did. You will read and answer the question and then let me know when you’re done.

Once you are done answering, I will ask you to look back at the question and the answer you chose and try to say out loud the thoughts that were in your head at each moment. If I don’t hear you speaking I’ll ask you to keep talking. I’m telling you that so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think aloud if you get quiet because I need to hear all of the thoughts you had during the task.

Okay, now go ahead and start working on the question. Let me know when you are finished.

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

[When students indicate they are finished, make sure they have circled two answers, and then ask them to begin describing their thoughts out loud. As necessary, include the appropriate prompting questions, such as the following:]

Okay, so tell me what you were thinking as you were reading and answering this question. What did you think at the beginning, when you started to read the question? … And then what were you thinking? … And what else did you think about? … Anything else? … Can you tell me more about that? … etc.

III c. Retrospective Think-Aloud: Student Practice II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual task. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIIa].)

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. [Place example question in front of student.] Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking all the way through—so you should tell me what you were thinking as you worked through it. So, first of all, go ahead and do the question, and let me know when you have finished.”

As before, during the post task think-aloud phase, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, section IIe).

Which two of the following objects have the most similar properties? Circle the two that are most similar

A. Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about what you were thinking while you were doing the task? Is this something you feel okay about doing? [If students say yes:] Good, then let’s begin our study.[If students say no or appear to be hesitant or reluctant, ask them to say more about any reservations they have, and try to address their concerns or uncertainties in a supportive way. If students indicate they do not wish to continue or do not feel comfortable continuing, allow them to stop.]

III d. Retrospective Think-Aloud: Starting the Tasks

Now we will move on to the actual tasks. Remember, after you have finished doing all the activities and answering all the questions on a task, I will ask you to say aloud everything that you remember thinking while you were doing the task, and I may remind you to do that if you are quiet. This task should take about [indicate correct time] minutes. Remember, you will not be graded on what you do during the task. There is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the tasks better. I will also have a few questions after we have finished.

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you are doing the task and while you are thinking aloud afterwards.

Here is the first task. When you are ready, go ahead and start working.

III e. Retrospective Think-Aloud: Prompts and Questions after Completion of Task

Now I would like you to tell me what you were thinking as we review what you did for the task. We will be recording what you are saying as you think-aloud. Feel free to stop at any time. What you say will not be used to grade you. The information will only be used to help improve the task. I will play the task and your writing back from the beginning now, and I want you to talk about the thoughts you had as you were working. From time to time, I might pause the video, to give you enough time to explain what you were thinking at that point or if I need to ask you any extra questions. I will let you know when I am pausing the playback.

The interviewer plays and watches the recording of the task in real time with the student, and asks the student to describe aloud what he or she was thinking at various points in the task. The interviewer should pause the recording whenever more time is needed, making sure to say “I’ll just pause this for a moment” or “I am pausing the task here,” so that the student understands that the playback has been paused.

If the student is not verbalizing enough, interviewers should offer a verbal nudge to remind the student to keep talking, such as:

What was going on in your head at this time?
I see you were looking at the task [or screen/passage/chart/image]. What were you thinking?
You seemed to be thinking hard—can you tell me what was in your mind right then?

If the student says something about his or her thoughts but it is difficult to tell exactly what the student means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on….
Uh huh, okay, and what else were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you were thinking?

III f. Notes on Student Actions and Behaviors during Tasks

As a student is proceeding through the task, a screen-capture system will record evidence of his or her actions on the task (such as where a student taps, how he or she moves through the task, and whether the student has entered any text into response fields). During the session, the interviewer should take notes of any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or reaction to the task. The following are examples of such behaviors for interviewers to note.

Does the student express signs of confusion, boredom, or excitement?
Does the student use tools provided in the task, such as tapping on tabs or buttons to reveal additional or earlier information or digital notepads?
Does the student rapidly move through the screen or take his or her time?
Does the student spend a lot of time on a particular section?
Does the student seem discouraged?

These kinds of observations should be considered opportunities to prompt students once they are done working to recall what they were thinking. In general, interviewers should make notes of any places during the course of the student’s working that appear valuable for follow-up questions after the task is completed (see section IV).

IV. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post task questions will include the following.

One standardized post task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions that are task specific. These will be selected for each task by ETS staff prior to testing. Interviewers will receive the set of up to three questions for the specific task they are examining in each cognitive interview.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to this task.

Additional Questions: Task-Specific Issues

The purpose of the additional post task questions is to capture more information on issues such as student actions during the task, particular aspects of [civics, geography, or U.S. history] targeted in the task, and general reflections about the task. ETS staff will identify up to three questions for interviewers to ask in relation to each task. Interviewers will also be trained to generate post-task questions on the fly, for cases where they observe something that they judge to need follow-up probing. The following are some examples of questions that might be asked at this stage.

Did any words in the task confuse you?
What ways could we change the task to make it better?
Did the [images, videos, audios, etc.] for the task help you answer the questions asked?

V. Eye Tracking

Eye tracking is designed to provide data about where students place their visual focus while taking an assessment. Students will be asked to do tasks on a tablet while a camera monitors where they look.

Apparatus

An eye-tracking system is video-based system utilizing a high speed camera and an LED infrared light that illuminates the face (no more powerful than typical sunlight). An Infrared camera is used to capture eye movements. The remote system requires no physical contact with the student: an infrared video camera acquires an image of the eyes and calculates gaze location in real time. To allow free (but limited) head and body movements while maintaining accuracy of localization, a reflective sticker is placed on the forehead of the student during the eye-tracking session. ETS eye-tracking experts have used this particular system with adults and with children as young as three years old. It takes a very short time (five minutes) to set up and calibrate the equipment for each participant.

Procedure

First, students are asked to work through a task in a normal way, without interruption. During this phase their eye movements are unobtrusively recorded and all events on the screen are captured in real time.

Next, using a retrospective verbal protocol, students are asked to go back over the task a second time and attempt to reflect on, and describe out loud, what they were thinking as they went through the task. These verbalizations are recorded.

To help the student recall his or her thinking as it occurred, a video (screen capture) of the complete task, including the student’s own movements and responses, will be played back for the student. A moving cursor indicating their gaze patterns will be overlaid onto the screen images so that the student can see exactly where he or she was looking while doing the task the first time. Seeing their own eye movements serves as a prompt to help students reconstruct their thinking at each point in the task.

These videos can be speed up or slowed down by the facilitator as necessary to allow focus on specific parts of tasks, or to allow sufficient time for students to verbalize their thoughts completely before moving on to later task sections.

The sequence to be followed for eye tracking and the subsequent retrospective think-aloud is described below. Note that the eye-tracking calibration procedure may differ slightly for various hardware setups.

Setup and Calibration

The participating student is greeted by an ETS eye-tracking lab expert. The expert describes the study procedure and responds to questions or concerns the student may have. The student is then seated in front of the eye tracker and a sticker is placed on his or her forehead to assist the eye tracker in locating the eyes and to compensate for head movements. The system is then calibrated. The student is asked to watch a dot appearing at a number of different locations on the blank screen. The setup and calibration typically take about five minutes.

Eye-tracking Session

After a successful calibration, the student starts the assessment session, in which he or she goes through the assessment task on the tablet. Students are free to move their head and body during the assessment within the field of view of the eye-tracking camera.

Along with the eye-tracking data, the system also records the following information: key presses, finger movements on the screen, screen changes (as a video file), and audio (optional). The interviewer takes notes during the session about noticeable behaviors or events.

Cued Retrospective Verbal Report

After the student completes the assessment task, eye tracking is stopped. Depending on the design, a retrospective think-aloud may be used. To help remedy memory decay and to prevent fabrication, the screen-cast video is replayed with the student’s eye movements displayed via a moving cursor showing the location of the student’s gaze. As the student watches the playback, he or she is asked to report on his or her thoughts.

I am going to show you how you did the task by showing you where you were looking as you went through the task. This is done by showing you a video playback of the task. This video will show your eye movements, finger movements, and keyboard movements. The red cross (+) that you will see on the screen will show where you were looking as you worked through the task. The movements and keystrokes are those you made as you worked. We are interested in what was on your mind at that moment in the task and why your eyes were at a particular spot in the task?

We want you to think about what was happening at that time shown in the video. For example, “Here, I was reading the question and then it looks as though I moved to the part where the answers were and looked over the options.”

Try not to just reread the directions or your answers. Instead, talk about your goals and plans at the time you were working on that part of the task. Tell us what was in your mind as you read, answered questions, and thought about the task.

To help students with the verbal protocol task, they are informed that they will be asked to speak continuously and that if they stop talking, they will be prompted with terms such as “please keep talking” and “please tell me more about that.” Students’ verbal reports are recorded for later coding and/or transcription.

VI. Debriefing and Thank You for Cognitive Interview

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions.]

Thank you for helping us to improve our test.

Thank the student for his or her time. Provide a gift card.

Part D. Small-Scale Tryout Protocol (Social Studies)

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word-for-word but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.

Session Information

The interviewer should complete the information from rows 1-3 prior to the beginning of the tryout session. Rows 4-5 are to be completed at the end of the tryout session.

Ref#	Preliminary Information	Data
1	Name of interviewer
2	Date
3	Start time of interview
4	End time of interview
5	Special circumstances that may have affected the tryouts session [Do not ask test taker, just record observations.]

Introduction Script:

Hello, my name is ______ and I work for EurekaFacts. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new social studies test questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out a task [and/or] small set of new test questions that are being developed for 8^th/12^th graders all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This tryout session is being recorded so researchers can review it later. What you say will not be told to anyone, or used in any way that could identify you, for any other purpose except as required by law [if needed: Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. We will look at what you say later, but only so that we can understand how our test is working and how we can make it better. Overall, this session should take about [indicate correct length] minutes.

We will work together to do one task [and/or] set of questions.

Do you have any questions?

Interviewer: If a student is no longer interested in participating, thank the student for his or her time and end the tryout session for that one student only: continue with all others.

Task Instructions

Ok, now, I’m going to give you a [task and/or set of questions] to complete.

This is a timed activity, which will last for [indicate length of activity], and everyone will begin at the same time.

All of the instructions will be displayed on your computer screens.

The [task and/or set of questions] you are about to do is something that students, like you, will do on the computer. However, we aren’t finished making them yet, so there may be some parts that don’t work. Try to work through and complete the [task/questions] as best as you can.

You will not be graded on what you do during the task, but please answer these questions as if it was a real test.

We will all work at the same time, first I would like you to read the instructions, once everyone has gone through the instructions, we will begin the [task and/or questions] together. I will start the timer and I will stop you once time is up.

If you finish early, please sit back and wait until the time is up. Wait until everyone is done and then we will submit your answers together.

Okay, I will come around and open the [task/questions] for each one of you, but please do not begin until I say that you may start.

Okay, now that everyone is ready, you may begin the task.

START THE TIMER NOW. STUDENTS HAVE XX MINUTES TO COMPLETE THE TASK AND/OR QUESTIONS. STOP STUDENTS AT XX MINUTES EVEN IF THEY HAVE NOT FINISHED.

Wait either for all students to complete the task and/or questions OR for the full amount of time to be up.

Once students have completed the task and/or questions, make sure that all students exit the task and submit their answers.

Debriefing and Thank You

Shape1

Before we finish, I’d like to hear what you all thought about what you’ve been doing.

Is there anything else any of you would like to tell me about working on the test questions?

Did you have any problems understanding [any of the questions/anything]?

Did you have any problems understanding what you were supposed to do to answer any of the questions?

Is there anything that you think could make [this/these] test question[s] clearer?

Is there anything you would like to ask me about what we did today? [Answer students’ questions.]

Thank participant(s) for their time and escort them to the front desk. Provide gift card(s).

1 60 minutes for play testing and tryouts, 90 minutes for cognitive interviews

2 Morae^® software (by TechSmith) can capture a student’s interactive behaviors as they happen, while one or more observers can simultaneously record comments that are time‐locked to the student actions and to the video recording. Adding Morae^® Observer software allows observers to be located in a remote location. This is both a convenience for observers and a potential means of reducing student stress or distraction.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-27