National Assessment of Educational Progress (NAEP) Science Scenario-Based Tasks Cognitive Interviews

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters located in Appendices A-H.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 120 minutes, including the time to review instructions and complete and review the information collection. If you have suggestions for improving this collection or any comments or concerns regarding the accuracy of the time estimate or the status of your individual submission of this form, please write to: National Assessment of Educational Progress, National Center for Education Statistics, Potomac Center Plaza (PCP), 550 12th St., SW, 4th floor, Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803

I. Cognitive Laboratory Script – Science Scenario-Based Tasks

Text written in italics is to be read aloud by the interviewer.

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate.

Introduction:

Hello, my name is ______ and I work for American Institutes of Research. It’s nice to meet you and thank you very much for coming today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?
- If the student responds with “science” or “computer lab”, ask them what they are studying and follow up with: Good, then I think you’ll enjoy what we are going to do today.
- If the student responds with another subject, respond accordingly (e.g., if the student responds with “English,” ask them what they are studying in their English class and why they enjoy it).

Let me begin by explaining why I am here and what you are going to be doing. You are participating in a special study about grade 8 science tasks, which are part of the National Assessment of Educational Progress, or NAEP for short. NAEP is a test given to students in grades 4, 8, and 12 in the United States. During the interview you will take one of the grade 8 science tasks, and then we will talk about your experience. The information we get from this interview will help us understand tasks like this so that we can make them better for other students like you.

It’s okay if you do not know how to do the science task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the task, so please try to do your best.

If at any time you decide you do not want to go on, you may stop doing the task and leave.

This interview is being recorded so we can review the recordings later. What you say may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

This new test is on the tablet computer. First, we will watch a short video with some instructions on how to use the controls and tools given to you. After the video, you will do a science task for [15 or 30] minutes. Then we will go back to a recording of the work you just completed and you can explain to me what you were thinking. I’ll close our session by asking you a small number of questions regarding your past experiences with computers and science learning activities that might be like what you have experienced here today.

Do you have any questions before we start?

Ask the participant if he/she has any questions. After answering questions and giving further explanation, continue. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

OK, let’s begin.

II. Retrospective Think-Aloud

IIa. Retrospective Think-Aloud: Instructions and Modeling Script

Using the retrospective (versus concurrent) think-aloud approach, students are instructed to first complete a task as they would under normal testing conditions (silently, without interruption). Then, after the task is completed, they will be asked to describe what they were thinking while reviewing the task on the computer. A replay of the task using Camtasia® software will help cue the student’s recollection of the thought processes that occurred during the task as the student progressed through it.

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, interviewers should be attentive to participants’ literacy for science content and digital technologies when delivering scripts.

Instructions:

To help us make our test better, we will ask you to complete a science task on the tablet computer. You will go through the task you will see on the screen, working at your own pace. Please note that there is a built-in time limit and that the task will end after [15 or 30] minutes.

While you are doing the task, we will be recording everything that happens on the screen. The screen recording will capture all of your actions and movements on the tablet. After you have finished doing the task on your own, I’m going to ask you to go back over the task with me, and I will ask you to tell me what you were thinking as you were going through the task. We call this “thinking aloud,” because we are asking you to say out loud everything you were thinking while you worked. To help you remember what you were thinking as you worked, we will look at the recording we made of you doing the task. This way, you will be able to see your screen actions and how you responded on the tablet computer.

As we watch the recording, I will also ask you some specific questions about what you think about different parts of the task. Please remember that we are especially interested in the visual and interactive features of the task. That is, the images you see on the screen and the actions you have to take to respond to the task on the tablet. Does that make sense? Do you understand what we’ll be doing?

Answer any questions and clarify the sequence of steps, if needed.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud. When I’m finished, I’ll ask you to try it.

Think-Aloud Demonstration:

Since we can’t tell what is going on in your head, we need you to “Think-Aloud.”

Let me give you an example.

Place example question in front of student.

Look at this question. It asks me to choose which two animals are the most similar. I’m going to do this question just as I would for a test. After I have finished, I will try to remember what I was thinking as I was working on the question so I can tell you.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

The interviewer ‘works’ silently for about a minute on the question. The interviewer should enact some behaviors that suggest that he/she is thinking hard about the question and carefully considering the possible answers. Some examples include: pointing a pencil tip at each option in turn; moving a pencil or fingertip between answer options; returning to the question; and giving nonverbal signals indicating consideration, uncertainty, etc.

Finally, the interviewer circles two answers, B and E, and then begins the retrospective think-aloud.

Okay, so here I was reading the question… It says I have to decide on the two animals that are the most similar. So at first I was wondering what do they mean by “similar?” That could mean anything. And I was thinking I wasn't sure how I’m going to make that decision. So... the choices... beetle, mouse, crab, dolphin, and cow. Yeah, well, as I'm reading the choices I'm already thinking there are some things about them that are similar, but there are also differences. I was thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But one lives in the water and the other doesn’t, so that makes them different. And the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Oh yeah, and then there is the dolphin that lives in the water, too… and I wondered about that, but that’s about all it has in common with the crab, so I couldn’t say they're very similar even though they’re both in the sea. So in the end I thought the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever… but then I thought, dolphins are mammals too, right? I think they are… anyway they seem so different from mice and cows to me. So I wasn't sure… it seemed like a hard question… I wasn't sure what to choose, but in the end I had to make a decision, so I decided to go for mouse and cow. Because they both have fur and four legs and walk around on land. It’s weird because they’re really not that similar. But, when I think of all the others, they seem really different from each other like in where they live, and their bodies. So even though it was really hard to decide, that seemed like the best choice to me, because they both live on land and they’re both mammals. So those things seemed pretty important and that’s how I made my decision.

Ok, let’s have you give it a try.

IIb. Retrospective Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he/she can read it. Allow the student as much time as needed to answer the question. When he/she indicates he/she has finished, ask the student to begin thinking aloud. Some students will be silent after reading the question, indicating they are finished. Such students should be asked to say whatever they are thinking. In general, it may be necessary to remind students to talk aloud during the retrospective think-alouds. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt students, being careful not to lead them. The interviewer needs to be familiar enough with the script to encourage the think-aloud in a natural, conversational manner.

Think-Aloud Practice:

Now you will try a think-aloud. I will give you a question like the one I just did. You will read and answer the question and then let me know when you’re done.

Once you are done answering, I will ask you to look back at the question and the answer you chose, and I want you to try to say out loud the thoughts that were in your head at each moment. If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that so you won’t think I am criticizing the way you are thinking. I’ll be reminding you to think-aloud if you get quiet because I need to hear all of the thoughts you had while reading and answering the question.

The interviewer hands the student the question at this point.

Okay, now go ahead and start working on the question. Let me know when you are finished.

Which two organisms below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

When the student indicates he/she is finished, make sure the student has circled two answers, and then ask him/her to begin speaking his/her thoughts out loud. As necessary, include the appropriate prompting questions, such as the following:

Okay, so tell me what you were thinking as you were reading and answering this question. What did you think at the beginning, when you started to read the question?
And then what were you thinking?
And what else did you think about?
Anything else? Can you tell me more about that?.

IIc. Retrospective Think-Aloud: Student Practice II (Optional)

Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual task. It may be beneficial to instead conduct another think-aloud demonstration for the student rather than doing another practice [see think-aloud demonstration in section IIa].

If the student struggles to think-aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for his/her first attempt regardless of how good it was, for example:

Very good—let’s do another one before we start the real task. Are you ready? Here is the next practice question.

Place example question in front of student.

Remember that after you have finished answering this question I will ask you to talk out loud to tell me what you were thinking while you were working. Go ahead and do the question, and let me know when you have finished.

As before, the interviewer should prompt the student to think out loud at any point during the post-task think-aloud phase whenever there are more than a few seconds of silence (see suggested prompts in section IIb).

Think-Aloud Practice:

Which two of the following objects have the most similar properties? Circle the two that are most similar

Silver coin

B. Chocolate coin

C. Gold coin

D. Blue plastic coin

E. Brown plastic coin

After the think-aloud phase is finished:

Now that you have practiced, do you feel that you understand how you should talk aloud about what you were thinking while you were doing the task? Is this something you feel okay about doing?

If the student says yes: Good, then let’s begin our study.

If the student says no or appears to be hesitant or reluctant, ask him/her to say more about any reservations he/she has, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.

IId. Retrospective Think-Aloud: Starting the Tasks

eNAEP Instructions:

Now we will watch a short video on the tablet that explains how to move around within the task, and how to use the tools that are available to you.

Interviewer plays the shorted eNAEP tutorial to orient the participant to specific features of the platform on which the SBTs will run.

OK. Now let’s move on to the actual science task. Remember, after you have finished, I will ask you to say aloud everything that you remember thinking while you were working on the task, and I may remind you to do that if you are quiet. I will also be occasionally asking you specific questions about the task features as the video plays. This task is timed and will automatically end after [15 or 30] minutes have passed. Remember, you will not be graded on what you do during the task. There is no right or wrong way to think-aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the task better. I will also have a few questions after we have finished.

Do you have any questions before we go on?

Answer any questions the student may ask.

As a reminder, you can tap the screen or use the mouse pad to interact with the task. Use the keyboard to type your answer when needed.

If you are ready, I will start recording and load the task for you.

At this point, the interviewer will begin the screen capture recording and load the task.

Alright, you may start when you are ready.

IIe. Retrospective Think-Aloud: Prompts and Questions After Completion of Task

Think-Aloud Instructions:

Now I would like you to tell me what you were thinking as we review what you did for the task. We will be recording what you are saying as you think-aloud. Feel free to stop at any time. What you say will not be used to grade you. The information will only be used to help improve the task. I will play the screen recording of the task and your actions back from the beginning now, and I want you to talk about the thoughts you had as you were working. From time to time, I might pause the video, to give you enough time to explain what you were thinking at that point. I will also pause if I need to ask you any extra questions. I will let you know when I am pausing the playback.

The cognitive lab interviewer gives the student a notebook containing still images from the task they completed.

Here is a notebook that has still images (or screenshots) from the task that you can also use to help you remember what happened throughout the whole task.

At this time, the interviewer brings out their own notebook containing screenshots and probe questions for specific points throughout the task.

Before continuing, interviewers will be prompted by their notebook to ask a set of introductory orienting questions, customized for each Science SBT, such as:

In your own words, please summarize what the task was about. What problem were the scientists trying to solve and how did they go about solving it?

The interviewer then plays and watches the screen recording in real time with the student. They are both following the progress of the task with their respective notebooks, and the interviewer asks the student to describe aloud what he/she was thinking at various points.

The interviewer should pause the recording whenever more time is needed, making sure to say either:

I’ll just pause this for a moment; or
I am pausing the task here.

This way, the student understands that the playback has been paused.

If the student is not verbalizing after 15 seconds, interviewers should offer a verbal “nudge” to remind the student to keep talking, such as:

What was going on in your head at this time?
You seemed to be thinking hard at this point in the task—can you tell me what was in your mind right then?
I noticed you [insert action of interest]. What were you thinking?

If the student says something about his/her thoughts but it is difficult to tell exactly what the student means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate, for example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on…
Uh huh, okay, and what else were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable, the interviewer can ask:

Can you explain what you mean by that?
Can you say a bit more, to help me understand what you were thinking?

In addition, the interviewer’s notebook will contain probes customized for each SBT task that ask students about specific task features as they come up in the replay, and are anticipated to have a facilitating or hindering impact on student performance. The probes will ask about the student’s reaction to the feature and, if relevant, how they think it might be improved.

IV. Student Background Questionnaire

After completing the cognitive interviews, the students will be administered a brief written survey regarding their past experiences with computers and science learning activities, and their attitudes toward digitally based task and science learning activities. The questionnaire is shown below.

Student Questionnaire

Section 1.

Do you have science class this year/this semester?

Have you done any scientific investigation in any of your science classes like the task that you were presented here?

If your answer is Yes, please describe the scientific investigation(s) in a few words.

In this school year, have you participated in any of the following activities? Fill in one oval on each line.

5. In this school year, have you visited a museum, zoo, or aquarium to learn about science?

6. How hard was this test compared to most other tests you have taken this year in school?

Section 2.

In this section, you will answer questions about different kinds of computers, tablets and smartphones.

This is an example of a desktop computer.

7. Do you have a desktop computer at home?

8. In this school year, how often have you used a desktop computer for science learning? Count all the times you did this at home, at school, in an after-school program, or anywhere else.

This is an example of a laptop computer.

9. Do you have a laptop computer at home?

10. In this school year, how often have you used a laptop computer for science learning? Count all the times you did this at home, at school, in an after-school program, or anywhere else.

This is an example of a smartphone. A smartphone is any phone that is able to connect to the Internet. Besides making phone calls and taking pictures, smartphones allow you to do many of the same things as a desktop or laptop computer.

11. Do you own a smartphone?

12. In this school year, how often have you used a smartphone for science learning? Count all the times you did this at home, at school, in an after-school program, or anywhere else.

This is an example of a tablet. A tablet is bigger than a smartphone. It allows you to do many of the same things as a smartphone and a laptop, but it does not make phone calls. If you use a tablet at school, you might use it to do things such as reading books or practicing math problems.

13. Do you have a tablet at home?

14. In this school year, how often have you used it for science learning? Count all the times you did this at home, at school, in an after-school program, or anywhere else.

15. At home, do you have Wi-Fi or some other Internet connection you can use?

16. Were you taught any of the following at school? Fill in one oval on each line.

17. Which best describes the way you type on the computer keyboard?

18. Compared to other students in your English/language arts class, how fast do you type on a computer keyboard?

19. Would you rather take a test at school using paper and pencil or using a computer?

20. How often do you feel you can understand what the teacher talks about in science class?

21. How often do you feel you can do a good job on your science tests?

22. How often do you feel you can do a good job on your science assignments?

23. For each item, please circle only one answer that best represents your opinion or belief.

24. Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in one oval on each line.

V. Debriefing and Thank You for Cognitive Interview

Thank the student for his/her time and provide the incentive.

Closing script:

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today?

Answer the student’s questions.

Thank you for helping us to improve our test.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	David_Work
File Modified	0000-00-00
File Created	2021-01-27

National Assessment of Educational Progress (NAEP) Science Scenario-Based Tasks Cognitive Interviews

NCES Cognitive, Pilot, and Field Test Studies System

Vol II - NAEP SBT Visual Effects Cog Labs