National Assessment of Educational Progress (NAEP) 2022 Economics Pretesting

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60-90 minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

Authorization and Confidentiality Assurance

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622).

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1850-0803 Approval Expires 07/31/2019

Cognitive Interviews Protocol

I. Introduction to Study Script

The interviewer should not read the script word-for-word but should be familiar enough with its content to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular item set being studied. In general, protocols will be tweaked and customized in light of specific item set content and greater knowledge of delivery devices.

Introduction:

Hello, my name is ______, and I work for [EurekaFacts]. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with two or three questions to get the student used to talking, such as:

That’s interesting—why do you enjoy [subject] so much?
What are you studying in [subject] at the moment?
And what’s the best thing you have studied in [subject] so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out what students know about economics. It is part of something called the National Assessment of Educational Progress, or NAEP for short, which is conducted by the National Center for Education Statistics, within the U.S. Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us develop better tests for other students. Overall, this session should take about [indicate correct length] minutes.

It’s okay if you don’t know how to do any parts of the [item set]. I will not be grading your work today, and your answers will not be shared with your parents or school. My goal is to learn how you react to the [item set], so please try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This new test is on the tablet. The instructions about how to do the test are also on the tablet.

This interview is being recorded with audio so we can review it later. We will also be able to capture what you tap on and where you move your fingers. What you say may be used only for statistical purposes to improve test questions and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [If needed, reference: (20 U.S.C. §9573 and 6 U.S.C. §151)]. We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together on [specify number of item sets].

Do you have any questions?

After answering any questions and giving further explanation, continue with the think-aloud training. If the student is no longer interested in participating, thank the student for his or her time and end the interview.

Okay, let’s move on. Before we look at the real [item sets], I want to make sure you have a good idea of what we’re going to be doing. So, I am going to give you some practice questions. The practice questions should help you get used to what we will be doing during the real [item sets]. They should help you understand how we want you to respond. Do you have any other questions before we start?

II. Concurrent Think-Aloud

IIa. Concurrent Think-Aloud: Instructions and Modeling Script

Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word-for-word but should be familiar enough with its content to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students when delivering scripts. Text placed in brackets is generic text that should be tailored to suit the particular item sets being studied.

To help us make our test better, we will ask you to complete some [item sets]. While you are doing the [item sets], we will be recording everything that happens on the screen. The screen recording will capture all of your responses and movements on the tablet, such as what you tap on and anything you put into text boxes.

I’m going to ask you to do the [item sets] in a way that may be different from what you are used to. Instead of working quietly, I want you to tell me what you are thinking as you work through the questions. We call this “a think-aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of the think-aloud process. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud. This will help you see how I want you to describe what you are thinking as you are working on the [item sets]. When I am finished, I’ll ask you to try it, so you can see how it works.

Think-Aloud Demonstration:

Since we can’t tell how you are thinking about a question while you are working on it, we need you to think aloud.

Let me give you an example. (Place example question in front of student.) Look at this question. It asks me to look at the five animals and choose two that are the most similar. So, I am going to do this question, and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar.

Beetle
Mouse
Crab
Dolphin
Cow

The interviewer should not preform the demonstration by reviewing the item and answering the question silently. Instead, the interviewer will verbally talk through their thought process as they read the item and consider the question.

Okay, so I am reading the question…It says I have to decide on the two animals that are the most similar. Hmm, I am just wondering what they mean by “similar.” That could mean anything. I’m thinking I’m not sure how I’m going to make that decision. Well, let’s look at the choices. So, the choices are beetle, mouse, crab, dolphin, and cow. Well, as I am reading the choices I am already thinking that there are some things about them that are similar, but there are also differences. I’m thinking the beetle and crab are kind of similar, in a way—well, they both have lots of legs and they kind of move around fast. But, one lives in the water and the other doesn’t, so that makes them different. I guess the mouse also moves around fast, but it’s not similar to either the beetle or the crab, except they are all small. Then there is the dolphin that lives in the water, too… but that’s about all it has in common with the crab, so I wouldn’t say they are very similar even though they’re both in the sea. Hmm, so the mouse and the cow are both mammals, they both have fur and they have live babies instead of laying eggs or whatever…oh right, but dolphins are mammals too, aren’t they? Yes, I think they are…but really, they seem so different from mice and cows to me. I don’t know…this seems like a hard question…I’m not sure what to choose. Okay, I’m going to try to make a decision…umm…okay, I think I am going to go for…mouse and cow. Because they both have fur and have four legs and they both walk around on the land. That seems weird; they are not really similar! But when I think about all of the others they seem too different from each other in really basic ways, like where they live and their kind of body and that kind of thing…So, yes, even though it was really hard to decide, that seems like the best choice to me, because they are both mammals and they both live on the land. So, those things seem like they are important, and that is how I made my decision.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do as you are thinking about the [item sets] today. The point of the think-aloud is to get at whatever is in your head as you are doing the [item sets]. Just say aloud the words and the thoughts that are in your head, as you are thinking about and working on the [item sets].

IIb. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so he or she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he or she works through the questions and tasks. If necessary, interviewers should use the “think-aloud hints” given below to prompt the student, being careful not to lead the student. The interviewer needs to be familiar enough with the information to introduce the think-aloud in a natural, conversational manner.

Now you will try a think-aloud. You can use this example (shown below). Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to talk out loud, telling me all of the thoughts that are going through your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now, so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet, because I need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Question:

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the [item sets]? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions, say: Good, then let’s begin our study.] [If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him or her to say more about the concerns or questions, and try to address the concerns or uncertainties in a supportive way. If the student indicates that he or she does not wish to continue or does not feel comfortable continuing, allow him or her to stop and say: Alright, let’s stop. Thank you for working with me today. Let me walk you out. Escort the student out of the session.]

IIc. Concurrent Think-Aloud: Student Practice II (Optional)

Note: Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may also be beneficial to model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa].

If the student struggles to think aloud, the interviewer should give the student another opportunity to practice. The interviewer should praise the student for the first attempt regardless of how good it was. For example: “Very good—let’s do another one before we start the real [item sets]. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through answering it. Tell me what you are thinking as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Question:

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the [item sets]? What questions would you like to ask me? [If the student says he or she feels OK and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he or she has questions, or appears to be hesitant or reluctant, ask him or her to say more about the concerns or questions, and try to address his or her concerns or uncertainties in a supportive way. If the student indicates that he or she does not wish to continue or does not feel comfortable continuing, allow the student to stop, and say: Alright, let’s stop. Thank you for working with me today. Let me walk you out. Escort the student out of the session.]

IId. Concurrent Think-Aloud: Starting the Set of Items/Tasks

Now we will move on to the actual [item sets]. Remember, as you do all the activities and answer all the questions, I’d like you to say aloud everything that you’re thinking. I may remind you to do that if you are quiet. These [item sets] should take about [indicate correct length] minutes. Remember, you will not be graded on what you do during the [item sets], and there is no right or wrong way to think aloud, as long as you keep telling me your thoughts. Your thoughts will help us make the [item sets] better. I will also have a few questions after you have finished working on the [item sets].

Do you have any questions before we go on? [Answer any questions the student may ask.]

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Here are the [item sets]. When you are ready, go ahead and start working.

IIe. Concurrent Think-Aloud: Hints for the Interviewer

The goal of think-alouds is to capture all the student’s mental processes while working through the item sets. Interviewers must strive to have the student speak aloud all of his or her thoughts while doing the item sets. If a student is continually providing short utterances or not talking, interviewers should use “continuers” to encourage students to be more descriptive, trying to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately five to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate on his or her responses.

If the student is not verbalizing enough, interviewers should offer a verbal nudge to remind the students to keep talking, such as:

What’s going on in your head right now?
I see you’re looking at the [item or screen/figure/chart/text]. What are you thinking?
You seem to be thinking hard—can you tell me what’s in your mind right now?
What are you thinking now?
Any other thoughts?

If the student says something about his or her thoughts but it is difficult to tell exactly what he or she means, or if the student begins to talk but does not say very much, interviewers can urge the student to elaborate. For example:

Tell me a bit more about that.
That’s interesting—can you say more about that?
Go on…
Okay, and what else are/were you thinking?

If more clarity or a more explicit description of a student’s thoughts is desirable:

Can you explain what you mean by that?
Can you say a bit more to help me understand what you mean/are thinking/were thinking?

It is important to be responsive and sensitive to each student’s behavior. If it becomes clear that students cannot say any more about their thoughts, interviewers should not keep asking. Students should be encouraged to think aloud as much as they can, but their reactions should be closely observed and pushing them should be avoided if students seem frustrated or uncomfortable. While it is desirable that students articulate as much as they can, sometimes students will simply be unable to state what is in their mind beyond a basic reply such as “I am reading the question.” Interviewers must be sensitive to nonverbal signals that may indicate students cannot say any more than they have or that they need a few extra seconds to gather their thoughts to put them into words.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this has occurred.

IIf. Notes on Student Actions and Behaviors During Set of Items/Tasks

As a student is proceeding through an item set, a screen-capture system will record evidence of his or her actions on the item set (such as where a student taps the screen, how he or she moves through the content and the questions, and any text or numbers entered into fields or boxes). During the session, the interviewer should take notes regarding any of the student’s expressions or behaviors that may reflect the status of the student’s understanding, engagement, or use of the item set. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
Does the student use the tools provided in the item set by tapping on tabs or buttons to reveal additional or earlier information, digital notepads, or reminders of what to include or think about?
Does the student rapidly move through the screen or take his or her time? Does the student spend a lot of time on a particular aspect?
Does the student interact with the task or interface in a new or unexpected manner?

These kinds of observations should be considered opportunities to prompt students to think aloud and to encourage them to express what is in their minds. Interviewers should also make a note of any places in the item set that appear valuable for follow-up with some additional questions after the item sets are completed (see Section IV).

III. Post-Think-Aloud Follow-Up Questions

After completing the think-aloud process for an item set, interviewers will follow up with a brief period of focused retrospective questioning. The post-think-aloud item set questions will include the following:

One standardized post-item set question that all students will be asked following all item sets, which is designed to discover whether the student has prior knowledge of the content.
Up to three additional targeted questions that are item set specific. These will be selected for each item set by ETS staff prior to testing. Interviewers will receive the set of up to three questions for the specific item set they are examining in each cognitive interview.
A set of approximately seven standardized post-item set economics multiple-choice questions that all students will be asked following all item sets.
Up to six survey questions that will provide information on topics such as digital resource use and/or current and prior experiences with assessments.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to [this item set] in school, or have you learned about or come across these things in your own life? [If yes: Tell me about what you have learned or studied or experienced that is related to [this item set].]

Additional Questions: Task-Specific Issues

The purpose of the additional post item set questions is to capture more information on issues such as student actions during each item set, particular aspects of economics targeted in the item set, and general reflections about the items. ETS staff will identify up to three questions for interviewers to ask in relation to each item set. Interviewers will also be trained to generate post-item set questions on the fly, for cases where they observe something that they judge to need follow-up probing. The following are some examples of questions that might be asked at this stage:

Did any words in the [item set] confuse you?
What ways could we change the [item set] to make it better?
Did the [tools, images, videos, audios, etc.] for the item help you answer the questions asked?

Additional Questions: Standardized Post Task Economics Questions

We wish to also gauge the student’s approximate economics knowledge with a few targeted economics questions. Students may be asked seven additional economics questions following the item sets designed to discover the student’s approximate ability in economics. The set consists of approximately seven brief multiple-choice economics-specific questions and takes approximately five to 10 minutes for students to complete. These questions will be drawn from materials already used in prior NAEP assessments. An example from NAEP-released items for Grade 12 is:

Which of the following best describes why the price of wheat might rise following a poor wheat harvest?

A. The demand for wheat has decreased.

B. The demand for wheat has increased.

C. The supply of wheat has decreased.

D. The supply of wheat has increased.

Additional Questions: Survey Questions

Up to six survey questions will provide information on topics such as digital resource use and/or current and prior experiences with assessments. Examples include:

On a weekday, about how many hours do you use a tablet, laptop, or desktop computer for doing schoolwork, including homework?

Response choices: None, Less than an hour, 1 to 2 hours, 2 to 3 hours, 3 to 4 hours, More than 4 hours

Have you ever worked on social studies or economics assignments on a tablet or computer before?

Response choices: Yes, No, Don't Recall

Overall, would you say it was very easy, easy, difficult, or very difficult to complete the economics questions here?

IV. Debriefing and Thank You for Cognitive Interview

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the [items set]?

Is there anything you would like to ask me about what we did today? [Answer student questions.]

Thank you for helping us to improve our test.

Thank the student for his or her time and provide a gift card.

Small-Scale Tryout Protocol

Text written in italics is to be read aloud by the interviewer. The interviewer should not read the script word-for-word but should be familiar enough with its content to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text in brackets is generic text that will be tailored based on task specifics.

I. Session Information

The interviewer should complete the information from rows 1-3 prior to the beginning of the tryout session. Rows 4-5 are to be completed at the end of the tryout session.

Ref#	Preliminary Information	Data
1	Name of interviewer
2	Date
3	Start time of interview
4	End time of interview
5	Special circumstances that may have affected the tryouts session [Do not ask test taker, just record observations.]

II. Introduction Script

Hello, my name is ______, and I work for EurekaFacts. I am here with my colleagues [introduce colleagues]. It’s nice to meet you, and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new economics questions. This is part of something called the National Assessment of Educational Progress, or NAEP for short, which is conducted by the National Center for Education Statistics within the U.S. Department of Education. Today, we’ll be asking you to try out a small set of new test questions that are being developed for 12^th graders all over the United States.

If at any time you decide you do not want to go on, that is your choice and you may stop.

After you complete the set of test questions, we will ask you some follow-up questions during a discussion session. An audio recording will be made of the discussion session so researchers can review it later. What you say will not be shared with your parents or school. It will be used only for statistical purposes, to improve test questions, and will not be disclosed, or used, in identifiable form for any other purpose except as required by law [If needed, reference: (20 U.S.C. §9573 and 6 U.S.C. §151)]. We will look at what you say later, but only so that we can understand how our test is working and how we can make it better. Overall, this session should take about 90 minutes.

We will work at the same time to do one set of test questions.

Do you have any questions?

Interviewer: If a student is no longer interested in participating, thank the student for his or her time and end the tryout session for that one student only. Continue with all other students.

Task Instructions:

Ok, now, I’m going to give you a set of items to complete.

This is a timed activity, which will last for [indicate length of activity], and everyone will begin at the same time.

All of the instructions will be displayed on your computer screens.

The questions you are about to answer will be something that students, like you, will do on the computer. Try to work through and complete the questions as best as you can.

You will not be graded on what you do during the session, but please answer these questions as if it was a real test.

We will all work at the same time. First, I would like you to read the instructions. Once everyone has gone through the instructions, we will begin the questions together. I will start the timer, and I will stop you once the time is up.

If you finish early, please sit back and wait until the time is up. Wait until everyone is done and then we will submit your answers together.

Okay, I will come around and open the questions for each one of you, but please do not begin until I say that you may start.

Okay, now that everyone is ready, you may begin.

Start the timer now. Students have [xx] minutes to complete the questions. Stop students at [xx] minutes even if they have not finished.

Wait either for all students to complete the set of test questions OR for the full amount of time to be up.

Once students have completed the set of questions, make sure that all students exit and submit their answers.

III. Post-Tryout Follow-Up Questions

After the tryout test questions, the interface will provide some follow-up questions. The post tryout questions will be similar to the following:

Which of the following best describes your exposure in grades 9–12 to economics concepts and content like in the questions on this test?

I have taken a class in economics focused on the kinds of concepts and content I saw in this test.
I have taken a social studies class in which we spent part of the time on the kinds of concepts and content I saw in this test.
I have taken a business class in which we spent part of the time on the kinds of concepts and content I saw in this test.
I have taken a class in which we studied personal finance but not the kinds of concepts and content I saw in this test.
I have not taken any classes in which we spent time on the kinds of concepts and content I saw in this test.

Have you ever used a tablet (for example, Surface Pro, iPad, Kindle Fire) or computer with a touch screen before?
1. Yes
2. No
3. I don't recall.

Overall, would you say that using the tablet made it easier, harder, or not much different to take this test compared to a paper and pencil version of this kind of test?
1. Easier
2. Harder
3. Not much different

IV. Post-Tryout Discussion Session

At the conclusion of the tryouts, the moderator will conduct a group discussion of approximately 15 minutes with the students present. An audio recording will be made of the discussion. The moderator will ask the students a number of questions related to the test questions in general or to individual test questions. Questions may focus on targeted areas or moderators may generate questions on the fly for cases where they observe something that they judge to need follow-up probing. The moderator will hand out screenshots of the test questions at the beginning of the discussion session to remind students. The following are some examples of questions that might be asked at this stage.

What ways could we change the questions to make them better?
- Did any words in the task confuse you?
- Did you understand the directions?
- What did you find easiest in the task? What did you find hardest?
Did the [images, videos, audios, etc.] for the questions help you answer the questions?
Did you understand what we were asking in question #__?
Did you feel that you had enough time to answer all the questions?
Did you feel that the questions that used drag and drop or grids to fill in were more interesting than the regular multiple-choice or short answer questions?
- Did you think they were more difficult, less, difficult, or about the same?
Is there anything else any of you would like to tell me about working on the test questions?

V. Thank You

Thank participant(s) for their time and escort them to the front desk. Provide gift card(s).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Ed Molin
File Modified	0000-00-00
File Created	2021-01-20