Vol. II

Trends in International mathematics and science study (TIMSS 2023) Cognitive Interviews

Volume II

Protocols

REQUEST FOR OMB Clearance

OMB# 1850-0803 V.263

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

October 2020

Table of Contents

TIMSS 2023 Cognitive Interview Protocol 3

Thank You and Closing Instructions 16

TIMSS 2023 Cognitive Interview Protocol

Overview

Each interview session will use a virtual format to test out either one longer or two shorter sets of questions that are referred to as a Problem Solving and Inquiry set, or PSI set, for short. The PSI sets will be presented on Adobe Connect software as PDF storyboards that show how the PSI set will appear to students on a digital device in the field test. As presented in Table 1 below, an interview session is composed of 6 or 8 parts and should last no more than 1 hour. Students administered two shorter PSI sets (expected to take 15 minutes each) will complete parts 1-8. Students administered one longer PSI set (expected to take 30 minutes) will complete parts 1-5 and 8. Each PSI set is administered in a one-on-one think-aloud session. During this part of the interview, the interviewer conducts quiet observations of the session, records the session, and takes notes of the student’s verbal reporting.

Table 1. Overview of the Interview Session

Time (minutes) for short PSI session	Time (minutes) for long PSI session	Short PSI session	Long PSI session
10	10	1. Welcome and introduction	1. Welcome and introduction
		2. Think-aloud and Adobe Connect practice	2. Think-aloud and Adobe Connect practice
		3. Instructions for the test	3. Instructions for the test
15	30	4. Think-aloud session – PSI set 1	4. Think-aloud session – PSI set 1
10	20	5. Debriefing of the think-aloud session	5. Debriefing of the think-aloud session
15		6. Think-aloud session – PSI set 2
10		7. Debriefing of the think-aloud session
		8. Closing of the session	6. Closing of the session
Total: 60	Total: 60

Interviewer Instructions

The following scripts should not be read verbatim; rather, the interviewer should be familiar with the content in order to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation, as appropriate. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer. Text in carets < > means that the wording may change depending on context. The interviewer should project a warm and reassuring manner toward the student and use conversational language to develop a friendly rapport.

Please fill out Table 2 below before starting Part 1 of the interview.

Table 2. Background Information

Date
Name of interviewer
Student ID number
Student’s gender	 Male  Female
Student’s grade

Please keep in mind that your total interview time with the student should last no more than 1 hour. You are asked to record the time at the beginning of each part of the interview session. At the end of the interview session (either the short PSI session or the long PSI session), please fill out Table 3 below to record the beginning time of each part and to calculate how long each part took.

Table 3. Time Records for the Interview

Short PSI interview			Long PSI interview
Part	Begin time	Duration (minutes)	Part	Begin time	Duration (minutes)
1		Part 2-Part 1:	1		Part 2-Part 1:
2		Part 3-Part 2:	2		Part 3-Part 2:
3		Part 4-Part 3:	3		Part 4-Part 3:
4		Part 5-Part 4:	4		Part 5-Part 4:
5		Part 6-Part 5:	5		Part 8-Part 5:
6		Part 7-Part 6:
7		Part 8-Part 7:
8			8
		Total:			Total:

Part 1. Welcome and Introduction

***[Please note the time: ________ ]***

Hello, my name is <name of interviewer> and I work for the American Institutes for Research. It’s nice to meet you, and I thank you very much for joining online today. I have enabled your ability to use your microphone and webcam for this interview, so on the top menu you can use the drop-down arrows to select the microphone and webcam you would like to share during the interview. If you are having difficulties setting this up or if the volume is too low, I can help you troubleshoot.

Once all three icons are green and active, then I should be able to see, hear, and speak with you.

To parents (if present at the time of the interview): I would like to let you know that you can stay in the room, but please sit off to the side during the interview to limit distractions. At the end of the interview, I will ask you to please return on camera so I can discuss the process for how you will receive the gift cards.

[Make small talk to build rapport with the student by asking a question, such as:]

• What is your favorite subject in school?

• What have you been learning in school this week?

[When the student responds, follow up with one question to get the student used to talking. For instance:]

• That’s interesting—why do you enjoy <subject> so much?

Well, I think you’ll enjoy what we are going to be doing today. Let me explain.

You are here today for a special study for the National Center for Education Statistics, part of the Institute of Education Sciences in the U.S. Department of Education. This study is called the Trends in International Mathematics and Science Study, or TIMSS for short. TIMSS is a test of mathematics and science, which will be given on a digital device to students like you in different schools in the United States and in many countries around the world. The American Institutes for Research is conducting interviews with students to ask for your help in making the test questions better. 

Over the next hour, I will be asking you to use software tools to answer a group of the questions that may be included in the final version of the test, which will appear on a digital device. These items are confidential and are not to be shared. If you have a cell phone nearby, please place it to the side so you are not distracted. Please make sure to close all other windows on your computer. Let me know when you have done that.

Some of the questions may be easy to answer and some may not be easy to answer or to understand. This is fine. Your answers will not be considered right or wrong and I will not be grading your answers, but we need you to answer the test questions the best you can and as if you are taking a real test.

After you finish a question, we will talk about what you thought of it. For example, I want to know what you think about the instructions and the tasks. With your help, we hope to make the test better and clearer for students your age. Because the information you provide is so important, I am going to be taking notes while you are working and when we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today.] Also, this interview will be recorded so researchers can review the recorded video file later if necessary. All of your personally identifiable information, like your name or your school, will be kept confidential, or secret, and your responses will only be used for research.

Remember, I am here to get information from you, and I will not be grading your work. Would you like to be a part of this study?

[If student agrees, continue; if not, end the interview. Also make sure you have collected the completed Parental Consent form by this time.]

If at any time you want to stop, just let me know.

[If, for any reason, the student is no longer interested in participating, thank the student for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the student may ask.]

Part 2. Think-Aloud Practice

***[Please note the time: ________ ]***

As you start answering some mathematics and science test questions, we want to learn what you think about these questions and how they are explained. One way we do this is to ask you to “think aloud” throughout what you are doing. All I mean by “think aloud” is for you to tell me what you’re thinking as you go through the questions and how you think you would answer them.

This might feel a little strange at first, so let’s do a short “think-aloud” practice. I’m going to give you a simple question and ask you to think aloud as you decide on your answer. I may remind you throughout the interview to think-aloud as you are working, since it’s not usual for people to do this. [Open student “think-aloud question” document on the screen so child can view it.]

[Document says: How many windows are there in the house or apartment where you live?]

[Some students may be silent. Encourage the student to say whatever he/she is thinking and probe as needed for detail. Let the child know it is okay to estimate the number of windows. If the student appears to need additional help with the concept, suggest that you demonstrate thinking aloud. Use the same question and demonstrate the process.]

[When student completes the practice:] Thank you; that’s the kind of detail I’m looking for when you talk to me while reading and answering the questions.

[Interviewer will enable drawing privileges for the student by clicking on their name in Adobe Connect.]

Now, for the second practice task, I would like you to use the software tools on the left-hand side of your screen to draw the shape of the window in your home that is closest to you. You can do this by clicking the main square icon on the black bar on the left-hand side to then select either the square or line icons. [Model the steps of inserting a square and a line shape into the PDF being shared.]

[If the parent asks if they can help the child figure out how to use the Adobe drawing tools, please let them know that they can do so for the practice exercise period.]

[Document says: Please use the shape or drawing tools on the left to draw the window closest to you.]

[When student completes the practice:] Thank you for using the tools to draw the window closest to you.

[Document says: Please also use the text box feature on the left-hand side of your screen to click and type in your response to how many windows there are in the house or apartment you live in.

For your third and final practice task, please use the text box feature by selecting the “T” icon on the black bar on the left-hand side of your screen to click and type in your response to the question regarding how many windows there are in your house or apartment. Allow me to model this function first. [Insert a text box for the student.] Now you try.

[When student completes the practice:] Thank you for using the tools to insert the text box with your response to the first question.

After you have inserted a shape, line, or textbox, you can use the Undo and Redo functions (i.e., the curved arrows) on the bottom of the list to remove and restore the changes that you have recently made. Allow me to demonstrate this, then feel free to practice it if you would like. [Demonstrate the Undo/Redo functions for the student.]

Part 3. Instructions for the test

***[Please note the time: ________ ]***

As we just practiced, there are no right or wrong ways to think aloud. What I want to know is what you’re thinking about as you try to understand and answer a question. You do not need to say everything out loud, just the main points of how you go about answering a question: what you find confusing, what questions you have about what is being asked.

We will start the test shortly. As you answer the questions and think aloud, I will take some notes but not interrupt you, and we will discuss your thoughts after you are done with answering the questions. Also, I’m going to begin the recording now.

8th Grade Only You may use your own calculator for these questions. Do you need a calculator?

[If yes, instruct them on how to access the calculator application on their computer. If no, proceed. If student asks for the calculator later, it is fine for them to access it at that time.]

Do you have any questions before we start?

[If yes, answer any questions the student may have and proceed. If no, proceed.]

OK, let’s get started. [Start the recording].

[For students taking two short PSIs] You will be presented with two 15-minute sets of questions, and we will discuss them after the first set and also at the end. [For all students] Read the questions and use the software tools that we practiced to mark your answers in the space provided. Please answer them as best as you can. Let me know when you have finished all of the questions on a page, and I will turn to the next page. Once you have finished all of the questions, we will begin talking about the questions you answered. Remember to think aloud as much as possible while you read and move through the questions.

[For students taking two short PSIs] If you are not finished in about 15 minutes, I will stop you to begin our first discussion. [For students taking one long PSI] If you are not finished in about 30 minutes, I will stop you to begin our discussion.

You may begin now.

[Give the student no more than 15 minutes (or 30 minutes for longer PSI sets) to complete to each PSI set. Take notes in Part 4—and if applicable, Part 6 below—based on student think-alouds, paying particular attention to places where the student found problems or seemed confused.]

[Observations to Focus On:

1. Student perception of degree of challenge/difficulty (listen to think-alouds, as well as observe what students do in responding to a task)

2. Student engagement in the PSI (listen to think-alouds, as well as observe student’s ease or frustration with questions)]

[Troubleshooting: The interviewer can assist throughout if the student forgets how to use the tools or asks a question about using the software. These are examples of situations when the interviewer can provide technical support to the student:

1. If the student experiences technical difficulty during the interview, the interviewer should intervene or help the student choose a software tool.

2. If the student forgets which button is used for a textbox, the interviewer can point the student to the tool.

3. If the student clicks on something he or she is not supposed to use, the interviewer should intervene or provide help if asked.]

Part 4. Think-Aloud Notes & Interviewer Observations – First PSI

***[Please note the time: ________ ]***

Insert Storyboard Description Here for Interviewer Notes

Part 5. Probing and Debriefing Questions – First PSI

[When the student has completed answering the PSI set or when approximately 15 minutes (for a short PSI) or 30 minutes (for a long PSI) have passed, begin the discussion.]

***[Please note the time: ________ ]***

Thank you for working on those <insert PSI name> questions. Now, I have several things to ask you about your work.

[If the student does not complete the test question set, ask the following question.].

Do you think you would have been able to complete the questions if you had been given more time?

[As a priority, you should ask specific probing questions below based on issues that came up in the think-alouds. You can include the notes for specific probing questions with the think-aloud notes you took for each question in Part 4.]

Probing Questions

Below are examples of specific probes based on think-alouds and observations. Please show the specific items being probed to refresh the student’s memory. (Note to OMB: These are examples that have been used in previous TIMSS cognitive interviews. The wording highlighted in yellow would be changed to reflect the TIMSS 2023 PSI questions):

• Can you tell me more about what you were thinking when you were answering the question <creating a six-sided garden for Lily>?

• What did you do to help you answer the [question]?

• You said, “I don’t know what I’m supposed to do” (or something similar) when you were looking at [the question]. Can you tell me what you found confusing?

• Have you seen the word <perimeter> [or other words or concepts that appeared difficult for the student] in other books you have read?

• What do you do if you read a word [or there is some new information] and you are not sure of its meaning?

• Can you tell me how you answered this question? Did you use <approach A> or <approach B>?

If you have time after the probing questions, the debriefing questions are prioritized so that you should ask the questions below in the order in which they are listed.

Debriefing Questions

1. Overall, do you think this set of test questions was easy, medium, or hard or a mix? [Follow up with] Why? [If appropriate] If you chose a mix, can you explain which parts were easier or harder?

2. Were there any test questions that were confusing? [Follow up with] Which question(s) were confusing and how were those questions confusing? [Follow up with, if appropriate] Do you have any suggestions for making that part of the test better or less confusing?

3. Did you feel that you were easily able to understand what each of the questions were asking you to do? Across the set of questions, was any part more difficult to understand? [Follow up by asking which part of the question(s) was more difficult to understand and why]

4. What was your favorite part of this set of questions? [Follow up with] Why?

5. Do you have any other comments?

Thank you. [If interview is for two short PSIs: Let’s move to the next and final set of questions.]

[If interview is for one longer PSI, skip to Part 8]

Part 6. Think- Aloud Notes & Interviewer Observations – Second PSI

***[Please note the time: ________ ]***

Insert Storyboard Description Here for Interviewer Notes

Part 7. Probing and Debriefing Questions – Second PSI

When the student has completed answering the second PSI set or when approximately 15 minutes have passed, begin the discussion.

***[Please note the time: ________ ]***

Thank you for working on those <insert PSI name> questions. Now, I have several things to ask you about your work.

[If the student does not complete the test question set, ask the following question.]

Do you think you would have been able to complete the set of test questions if you had been given more time?

[As a priority, you should ask specific probing questions below based on issues that came up in the think-alouds. You can include the notes for specific probing questions with the think-aloud notes you took for each question in Part 6.]

Probing Questions

Below are examples of specific probing questions based on think-alouds and observations. Please show the specific items being probed to refresh the student’s memory. (Note to OMB: These are examples of probing questions that have been used in previous TIMSS cognitive interviews. The wording highlighted in yellow would be changed to reflect the TIMSS 2023 PSI questions):

• Can you tell me more about what you were thinking when you were answering the question <creating a six-sided garden for Lily>?

• What did you do to help you answer the [question]?

• You said, “I don’t know what I’m supposed to do” (or something similar) when you were looking at [the question]. Can you tell me what you found confusing?

• Have you seen the word <perimeter> [or other words or concepts that appeared difficult for the student] in other books you have read?

• What do you do if you read a word [or there is some new information] and you are not sure of its meaning?

• Can you tell me how you answered this question? Did you use <approach A> or <approach B>?

If you have time after the probing questions, the debriefing questions are prioritized so that you should ask the questions below in the order in which they are listed.

Debriefing Questions

1. Overall, do you think this set of test questions was easy, medium, or hard or a mix? [Follow up with] Why? [If appropriate] If you chose a mix, can you explain which parts were easier or harder?

2. Were there any questions that were confusing? [Follow up with] Which question(s) were confusing and how were those questions confusing? [Follow up with, if appropriate] Do you have any suggestions for making that part of the test better or less confusing?

3. Did you feel that you were easily able to understand what each of the questions were asking you to do? Across the set of questions, was any part more difficult to understand? [Follow up by asking which part of the question(s) was more difficult to understand and why]

4. What was your favorite part of this set of questions? [Follow up with why?]

5. Do you have any other comments?

Thank you.

[After the cognitive interview is complete or 60 minutes have passed, the interviewer should proceed to Part 8.]

Thank You and Closing Instructions

Part 8. Thank You for Participating

***[Please note the time: ________ ]***

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making the TIMSS test questions better.

[Ask if the child’s parent can join the video, and if so, confirm their e-mail address. Please send the Thank You E-mail along with the electronic gift card links via e-mail prior to ending the meeting with them and confirm they have received it. Once confirmed, thank them for their time and effort.

If the child’s parent is not available, please thank the child for their time and effort and let them know that you will be sending the Amazon gift card to their parent by email. After the interview, send the Thank You E-mail along with the electronic gift card links via e-mail to the email address the parent used when submitting their consent form.]

After the interview: Please go back to page 2 and fill out Table 3 to record the beginning time for each part and calculate how much time each part took.]

Think-Aloud Practice Question Document:

How many windows are there in the house or apartment where you live?

Please use the shape or drawing tools on the left-hand side of your screen to draw the window closest to you.

Please also use the text box feature on the left-hand side of your screen to type in your response to how many windows there are in your house or apartment.