National Assessment of Educational Progress (NAEP) Survey Assessments Innovations Lab (SAIL) Science Playtesting and Cognitive Interviews

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850–0803. The time required to complete this information collection is estimated to average 90 minutes, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this form, please write to: National Assessment of Educational Progress, National Center for Education Statistics, PCP, 550 12th St., SW, 4th floor, Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Playtesting Protocol

Script for Introduction to Study, Goals, and Approach

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the playtesting sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending the meaning of instructions or questions. Text placed in brackets is generic text that should be tailored to suit each of the research projects, as needed for the Virtual Science Lab.

Hi, our names are __ and we are from the Educational Testing Service. We are running a research project on a virtual, computer-based version of the science lab. We are doing this research for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. The system you’ll be seeing today isn’t part of the test right now, but we are doing this research to see how it works. Your input will help us make the system work as well as it can for all students. We are going to ask for your opinions based on some activities that we will ask you to do using the system. You are here to give us your thoughts about your experiences of interacting with the virtual science lab. We’d like to hear about what you think is good, what you think is not good, and if you think anything is difficult or confusing.

My colleagues and I will take notes on what you are saying—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts and by interacting with the system so we can see how it works when it is being used. There are no right or wrong thoughts or opinions or actions.

This process is being recorded so that we can review it later. Unless your parent/guardian gave permission, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). We will look at what you say later, but only so that we can understand how our system is working and how we can make it better. Overall, this session should take about <say length> minutes. If at any time you decide you don’t want to go on, that is your choice, and you may stop [If the student is no longer interested in participating, thank the student for his/her time and end the interview].

As you work through these activities, [FOR STUDENTS WORKING IN GROUPS] we want you to talk with each other as you go along [or FOR INDIVIDUAL SESSIONS] we want you to think aloud as you go along, so I’ll be asking you questions like “what are you thinking?” so you can tell me what’s in your head as you are thinking about what you are doing. We might also ask you a few questions now and then.

In Playtesting sessions, students will work either individually or in pairs/small groups with a facilitator (plus an additional observer in the case of individual sessions). The think aloud approach will be less formalized than in the cognitive interviews, and there will be more ad hoc questioning by the facilitator as needed (see below), so it doesn’t require a formalized practice session.

For the Virtual Science Lab study, students will be asked to complete tasks created around the manipulation of tools in the Virtual Lab. A facilitator will work with individual students or small groups and describe what students should do. Tasks will be based on principles of science inquiry. Examples of prompts are:

You will help us to test how easy it is to use the interactive tools in the virtual lab to accomplish some basic lab tasks, and how quickly you can perform tasks. We would like to you think aloud during your process of completing the tasks, which means we’d like you to say out loud all the thoughts that come into in your head as you work.

The first task you need to complete is to measure 5 grams of salt. Can you figure out where to find and select tools that you think are relevant to this measurement task, move the selected tools into the experiment environment, and then carry out the measurement task?

Once the facilitators have explained the activity and asked the students to begin, they will generally observe them with little interruption or minimal reactive influences on their thinking and will listen to individuals’ verbalized thoughts or small groups’ conversation for insights into what the students are thinking about the items and tasks. Intermittently during the activity, or following a given activity as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this difficult? Why?/Why not?

Are there any things that seem confusing here?

Do you understand what you need to do?

Do you think you can do what is needed here?

Have you done anything like this before?

How could this activity/the system be improved?

Were there aspects of what you just did that you didn’t like? Why?

This activity asked you to solve a problem about <insert focus of activity>. Have you learned about this before, either in school or anywhere outside of school?

Facilitators will observe students, take notes, and will pay attention to verbal answers to the questions above and write them down, but also will look for evidence of engagement, boredom, frustration, misunderstanding, or confusion in what students say to each other and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing playtesting sessions will introduce a question to individual students or groups of students talking about the activity to get more information from them, particularly in cases when students talk about issues related to their interest (or non-interest) in the activities, or confusing aspects of an activity or of the system. In such cases, observers might ask something like, “Can you tell me more about what you said?” If students do not provide sufficient comments on targeted parts, a staff member may ask individual students or a group of students if they had any thoughts about those aspects, using questions such as those described above, but focused on specific aspects or issues.

At the end, students will be thanked for their participation and for helping us with our research. Students will be given gift cards for their participation at the conclusion of each playtesting session.

Part C. Cognitive Interviews

I. Introduction to Study Script

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending meaning. Text placed in brackets is generic text that should be tailored to suit each of the research projects, as needed (Virtual Science Lab or Interactive Virtual Models). In general, protocols will be adjusted and customized in light of how individual students respond.

Introduction:

Hello, my name is ______ and I work for Educational Testing Service. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out how students interact with a [virtual science lab/virtual interactive model of the solar system/hybrid hands-on categorization task]. It is related to the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us test out these new systems and try out new activities that in the future may be used with other students. Overall, this session should take about [indicate correct length] minutes.

It’s okay if you don’t know how to do any parts of these activities. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the activities and to the system you will be using, so please just try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This interview is being audio and video recorded so we can review it later. Unless your parent/guardian gave permission, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). We will look at what you say and do later, but only so that we can understand how our test is working and how we can make it better.

We will work together on a few different activities today.

Do you have any questions?

If the student is no longer interested in participating, thank the student for his/her time and end the interview. After answering any questions and giving further explanation, the facilitator will conduct the think-aloud training.

Okay, let’s move on. I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice activities. The practice activities should help you get used to what we will be doing during the real tasks. They should help you understand how we want you to respond. Do you have any other questions before we start?

II. Concurrent Think-Aloud method

II a. Concurrent Think-Aloud: Instructions and Modeling Example Script

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students. Text placed in brackets is generic text that should be tailored to suit each of the research projects, as needed (Virtual Science Lab or Interactive Virtual Models).

To help us develop and improve our system, we will ask you to complete some activities. We will be audio and video recording you as you do the activities, and a separate recording will capture all of your responses and actions with the system.

I’m going to ask you to do these activities in a way that may be different from what you are used to. Instead of working quietly, I want you to say aloud whatever is in your head as you work through them. We call this “thinking aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of what it’s like to think aloud. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what is in your head as you are working on the activities. When I am finished I’ll ask you to try it, so you can see how it works.

Sample Think-Aloud Demonstration:

The example below shows how the facilitator should model the think aloud method for students. The key is to help them see that what is required is a fluid, naturalistic, continuous verbalization of all thoughts as they occur. Research staff may choose alternative tasks or activities as the basis for the demonstration, as appropriate. For example, since the Virtual Models study uses manipulatives in the form of hand-held controllers/natural objects, a physical manipulation-while-thinking task may provide a suitable example; since the Virtual Science Lab study uses an interactive tablet-based task, a tablet-based activity may provide a suitable example. ETS research staff will select an appropriate task or activity to demonstrate think aloud for each study, and facilitators will be trained to model and elicit this kind of naturalistic think-aloud data using the example.

Since we can’t tell what is going on in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

Two animals most similar. Beetle crab lots of legs… move fast. One lives in the water the other not… Mouse moves fast… but not similar, except small. The dolphin lives in the water, but that’s all… Mouse and cow… mammals, have fur, live babies not laying eggs… Others too different. So ‘mouse and cow’… mammals …on land…

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do today. The point of the think-aloud is to get at whatever is in your head as you are working. Just say aloud the words and the thoughts that come into in your head, as you go along. Don’t try to have a conversation with me or explain anything to me—just have a conversation with yourself, saying out loud anything that pops into your head as you go along.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so that he/she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt the student, being careful not to lead the student and using the hints as reminders to verbalize but in a non-intrusive way. Since the goal is to gain insights into student cognition as it occurs naturally during these activities, it is important to avoid questioning students or otherwise leading or pushing them in any particular direction. It is also important to help students understand that they should not attempt to explain their thinking to the interviewer, in fact they should not even think of the process as describing their thinking for someone else, but rather they should simply say aloud any words that come into their heads, even if they result in broken sentences and unfinished thoughts, so that the process of thinking aloud does not alter or interrupt the natural flow of the internal thinking process (Ericsson & Simon, 1989; see also Ericsson, 2006; Ericsson & Simon, 1984; Newell & Simon, 1972).

Now I want you to try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to say out loud all of the thoughts that come into your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because we need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II c. Concurrent Think-Aloud: Student Practice Script II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa])

If the student struggles to think aloud, or if the verbalizations are not of the right kind, the interviewer should give the student another opportunity to practice and help to provide guidance in terms of how to make their verbalizations more appropriate. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—say whatever pops into your head as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels ok and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II d. Concurrent Think-Aloud: Starting the Tasks

Now we will move on to the actual task. Remember, as you are working I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet. This part should take about [indicate correct length] minutes. Remember, you will not be graded on what you do and there is no right or wrong way to think aloud, as long as you keep saying any thoughts that you are having. Your thoughts will help us make the activities and the system better. I will also have a few questions after you have finished working.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Ready? OK, let’s begin.

The examples below illustrate the kinds of tasks and activities that students will be asked to work on during the cognitive interviews as they think aloud. During a cognitive interview session, a number of these kinds of activities will be given to each student, as time allows. If students become stuck or are visibly having difficulty, the facilitator will help them out with hints, or if a student is having a lot of difficulty, the facilitator can tell them that they have done what is needed and ask them to move on to the next activity or task.

Sample Virtual Science Lab Task Script:

To help us evaluate the Virtual Science Lab, we created a simple task that we would like you to carry out in the virtual lab. Your task is to measure 5 grams of salt. To complete this task in the Virtual Lab, you need to first select and find the relevant lab tools and materials. Then you will use the tool cart to deliver the tools and materials to the workstation to carry out the measurement. Remember to think aloud while doing this and pretend as if I’m not even here. If you fall silent for a while, I will remind you to keep thinking aloud. Are you ready? Then you can go ahead and start.

Sample Virtual Models Activity Script:

To help us evaluate the Virtual Solar System Model, we created a simple task that we would like you to carry out. To complete this task, first imagine you’re out in space, looking at the earth, and use the model to go to that view. Now use the controllers to figure out or show me why it’s hotter here in the summer than in the winter. Move the controllers to move the objects on the screen in a way that shows or explains what goes on in those different seasons. Remember to think aloud while doing this and pretend as if I’m not even here. If you fall silent for a while, I will remind you to keep thinking aloud. Are you ready? Then you can go ahead and start.

To help us evaluate the Hybrid Hands-On Categorization Task, we created a simple activity that we would like you to carry out. To complete this task, first read the instructions on the tablet and follow the steps to inspect the small natural objects on the desk in front of you and place each object into one of the containers. All of the objects belong to a single overall class, but you must categorize them into subsets based on your observation of their physical characteristics. The tablet will also display questions for you to answer as you work on the activity. Remember to think aloud while doing this and pretend as if I’m not even here. If you fall silent for a while, I will remind you to keep thinking aloud. Are you ready? Then you can go ahead and start.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of the think-aloud method is to capture all the student’s mental processes while working through activities. Interviewers must strive to have the student speak aloud all of his or her thoughts during tasks. If a student is not talking, interviewers should use “continuers” to encourage them to talk. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate his/her responses.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the students to keep talking, such as:

So what’s in your head?
What are you thinking?
What’s in your mind?
What are you thinking right now?
Go on….
Okay….keep going, keep talking

It is important to be responsive and sensitive to each student’s behavior. Students should be encouraged to verbalize as continuously as they can, but their reactions should be closely observed and pushing students should be avoided, especially if a student seems frustrated or uncomfortable. While it is desirable that students articulate as continuously as they can, sometimes students will simply be unable to say what is in their mind. Interviewers must be sensitive to nonverbal signals, if students cannot say any more than they have done.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors

As students are proceeding through each activity they will be video recorded. In addition, a screen-capture system and a video camera focusing on the tablet screen (in the case of the Virtual Lab and the Virtual Models hybrid hands-on studies) or digital log file capture (in the case of the Virtual Model studies) will record evidence of his or her interactions with the system or task. During the session the interviewer should take notes about any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task or system. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
How does the student use the tools provided?
Does the student rapidly move through the activity or take his/her time?
Does the student spend a lot of time on a particular aspect?

Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section IV).

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, based on the student’s behavior during the task, and based on time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

When you saw [information/graph/image] at this point, what were you thinking? What was in your mind?

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed on this part of the task that you did not use [describe tool or action]. Can you tell me more about why you didn’t use that?

I noticed that on this part of the task you spent some time looking at/going back-and-forth between [describe images, text, or actions]. Can you tell me more about what was going on at that point?

If students’ answers are unclear or not very explicit, the interviewer should use prompts to encourage them to say more, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting. Tell me more about that.

III. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post-task questions will comprise:

One standardized post-task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions. These will be generated by research staff prior to testing.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to what you did today.

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as student actions during the activities, particular aspects of knowledge or skills targeted in the activities, and general reflections about the activities.

The following list shows some sample question frames that research staff will use to build questions. The specifics of each question will be generated via an informal task analysis in which key points for understanding student thinking will be identified.

Let’s think about the part where you [describe key action or aspect]. Can you tell me more about what you were thinking when you were doing that?

This [button or other tool] was on the screen to allow you to [describe action]. Tell me about how and when you used that, or if you didn’t really use it, why was that?

IV. Debriefing Questions and Thank You for Cognitive Interview

Thank student for his/her time. Provide a gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on these tasks?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to study these processes and improve our system.

References

Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. The Cambridge handbook of expertise and expert performance, 223-242.

Ericsson, K. A., & Simon, H. A. (1984). Verbal reports as data. MIT Press.

Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity, 5(3), 178-186.

Newell, A., & Simon, H. A. (1972). Human problem solving (Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-Hall.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-27