Trends in International Mathematics and Science Study (TIMSS 2023) Cognitive Interviews

Students must be in 4th or 8th grade for interviews conducted through June 2020 and must have just completed 4th or 8th grade for interviews starting in July - November 2020
Students must have varied mathematics abilities
Students must have a mix of demographic characteristics: gender, race/ethnicity, socioeconomic status (of parents), and school type

STUDENT’S FIRST NAME:

PARENT’S FIRST NAME (SPOKEN TO / GIVES PERMISSION): _____________________________

DATE RECRUITED: RECRUITED BY:

DATE CONFIRMED: CONFIRMED BY:

ASK TO SPEAK TO THE HEAD OF THE HOUSEHOLD:

Hello, this is from EurekaFacts, an independent market research firm.

We are conducting a brief, but important study about education. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation. I’d like to tell you a little bit about the study first; is now a good time? (YES or NO and call back later)

We are looking for students to help us understand what they think about new interactive items for an international math and science study, called the Trends in International Mathematics and Science Study, or TIMSS [note: pronounced tims, like short for Timothy]. The study is conducted at regular intervals in the United States and many other countries around the world. TIMSS provides information that allows countries to understand the math and science knowledge of students globally. In the U.S., the National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the Department of Education, sponsors this study in collaboration with the American Institutes for Research (AIR). The purpose of the study is to find out what students think about the new items in order to help us make the study better and more understandable for 4th- and 8th-graders. The study would take place with a researcher at AIR offices in Crystal City, Virginia, or Rockville, Maryland, to discuss the new math and science questions for TIMSS. I’d like to ask you a few questions to see if you qualify for the study.

P1. Do you have a child in your household who is 18 years old or younger?

	SELECT ONE

Yes	1	®®(CONTINUE)

No	2	®®(THANK AND TERMINATE)

P2a. How old is this child?

RECORD AGE

P2b. What grade is your child currently in?

RECORD GRADE

(April–June 2020: MUST be in 4th or 8th grade. OTHERWISE, THANK AND TERMINATE)

(July–November 2020: MUST have completed 4th or 8th grade in the prior school year. OTHERWISE, THANK AND TERMINATE)

P3. What type of school does your child currently attend? (READ LIST)

	SELECT ONE
Public school	1	(ATTEMPT TO RECRUIT A MIX)
Private school	2	(ATTEMPT TO RECRUIT A MIX)

Homeschool Don’t know	3 4	(THANK AND TERMINATE)

P4. We are conducting a study with students about new items for a study on math and science.………If your child qualifies, do you give your permission for us to invite him or her to participate in a 60-minute interview to talk about these new items? $75 in total will be given at the end of the interview as a thank-you (gift cards of $25 will be given to the parent and $50 to the student), and parking will be validated at the interview site.

	SELECT ONE
Yes	1	(CONTINUE)

No	2	(THANK AND TERMINATE)

P5. From your perspective, on a scale from 1 to 4 (where 1 means “not at all comfortable” and 4 means “very comfortable”), how comfortable would your child be expressing his or her thoughts to a researcher he/she has not met before about whether a set of study questions and instructions presented to him or her are clear and understandable? (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3	(CONTINUE)

Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1	(THANK AND TERMINATE)

P6. Before your child participates in the research study, you must sign and return a parental consent form. Your child can either bring the signed form to the interview or you can scan it and e-mail it back. Without a signed form, your child cannot participate in the interview and receive the honorarium. May I e-mail you a parental consent form if your child chooses to participate?

	SELECT ONE
Yes	1	(CONTINUE)

No	2	(THANK AND TERMINATE)

The next few questions are for classification purposes only.

P7. How well would you say your child typically does in his/her math class? (READ LIST)

		SELECT ONE
	Outstanding (letter grade A)	1	→(ATTEMPT TO RECRUIT A MIX)
	Very good (letter grade B)	2

	Satisfactory (letter grade C)	3
	Needs improvement (letter grade D)	4
	Not achieving (letter grade F)	5
(DO NOT READ)	Refused	6

P8. Is your child Hispanic or Latino? (DO NOT READ OPTIONS)

	SELECT ONE
Yes	1	→(ATTEMPT TO RECRUIT A MIX)
No	2	→(ATTEMPT TO RECRUIT A MIX)
Prefer not to answer	3

P9. Which of the following categories best describes your child’s ethnic or racial background? (READ LIST)

	SELECT ALL THAT APPLY
American Indian or Alaska Native	1	(ATTEMPT TO RECRUIT A MIX)
Asian	2
Black or African American	3
Native Hawaiian or Other Pacific Islander	4
White	5
Other	6

P10. Is your total annual household income before taxes: (READ LIST)

		CIRCLE ONE
	Less than $15,000	1	(ATTEMPT TO RECRUIT A MIX)
	$15,000 to less than $30,000	2
	$30,000 to less than $45,000	3
	$45,000 to less than $60,000	4
	$60,000 to less than $75,000	5
	$75,000 to less than $100,000	6
OR	$100,000 or more	7

(DO NOT READ)	Refused	8

At this point, I’d like to be able to ask your child a few questions, during which time you are welcome to stay on the line. Is he/she available to speak with me now? (If not, get a re-call time and call back.) What is your child’s name? (RECORD NAME)

Again, you are welcome to stay on the line, but please allow your child to speak for him/herself. Thank you.

RECRUITER: Please determine how talkative and articulate the student is. It is critical that we speak with students who are comfortable expressing themselves with someone they are meeting for the first time. Please use your best judgment and recruit only students who are thoughtful and freely express themselves in a manner that is relevant to the topic being discussed.

All recruits MUST be articulate students who can answer open-ended questions and can provide support or explanation for their responses (for instance, “I like it because...” or “I don’t like it because...”).

DO NOT recruit any students who have difficulty answering these questions, whose parent answers FOR them, who seem hesitant or shy speaking with you (an adult they do not know), who answer “I don’t know,” or who CANNOT provide an explanation supporting their responses.

SAY TO STUDENT:

Hi (insert target student’s first name). My name is and I have been speaking with your parent for a few minutes.

As I explained to your parent, we are looking for your help in developing questions for a test about math and science. The test is called the Trends in International Mathematics and Science Study, or TIMSS, and is conducted all around the world. We are looking for students to tell us what they think about the new math and science questions so that we can make the test better. Right now, I’d like to talk to you to find out if you are able to participate in the study. There are no “right” or “wrong” answers to any of these questions, and whatever you say is OK. OK?

Great.

ASK THE FOLLOWING QUESTIONS OF THE STUDENT:

C1. What grade level are you currently in? (DO NOT READ LIST)

RECORD GRADE

(April–June 2020: MUST be in 4th or 8th grade. OTHERWISE, THANK AND TERMINATE)

(July–November 2020: MUST have completed 4th or 8th grade in the prior school year. OTHERWISE, THANK AND TERMINATE)

C2. What is the name of your school? (RECORD)

_________________________________________________________________________

C3. How well would you say you typically do in your math class? (READ LIST)

		SELECT ONE
	Outstanding (letter grade A)	1
	Very good (letter grade B)	2

	Satisfactory (letter grade C)	3
	Needs improvement (letter grade D)	4
	Not achieving (letter grade F)	5
(DO NOT READ)	Refused	6

C4. How comfortable do you think you would feel talking with a researcher you haven’t met before about how clear a set of questions and instructions are to you? Would you be: (READ LIST)

	SELECT ONE
Very Comfortable	4	(CONTINUE)
Somewhat Comfortable	3	(CONTINUE)

Not Too Comfortable	2	(THANK AND TERMINATE)
Not At All Comfortable	1	(THANK AND TERMINATE)

C5. Please tell me what your favorite subject is in school and why. (WRITE VERBATIM)





NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE- TO TWO-SENTENCE UNPROMPTED ANSWER IN WELL-UNDERSTOOD ENGLISH, PLEASE TERMINATE.

C6. RECORD GENDER OF STUDENT. DO NOT ASK.

	SELECT ONE
Male	1	(ATTEMPT TO RECRUIT A MIX)
Female	2	(ATTEMPT TO RECRUIT A MIX)

ASK TO SPEAK TO PARENT AGAIN:

(FOR STUDENTS WHO DO NOT QUALIFY): Thank you very much for your time, and we really appreciate your and your child’s interest, but unfortunately, your child does not qualify for this current project. Have a good morning/afternoon/evening.

INVITE TO SESSION

Your child qualifies to participate in an interview session with a researcher at AIR offices in Crystal City, Virginia, or Rockville, Maryland, to discuss the new math and science questions for TIMSS. The interview is scheduled for (INSERT DAY/DATE AND TIME). The session will last 60 minutes. The interview is strictly for research purposes. Let me assure you that no one will attempt to sell or enroll you in anything. At the completion of the interview, we will provide a $50 gift card for your child and a $25 gift card for the parent who provides the child with transportation to and from the interview.

Will you and [child’s name] be able to attend the interview?

	SELECT ONE
Yes	1	(CONTINUE)

No	2	(THANK AND TERMINATE)

In which AIR office do you prefer to have the interview take place?

	SELECT ONE
Crystal City, VA 1400 Crystal Drive 10th Floor Arlington, VA 22202	1
Rockville, MD 6003 Executive Blvd. 3rd Floor Rockville, MD 20852	2

To attend the session and receive your honorarium, you must bring with you the signed “PARENTAL CONSENT” form that we will e-mail you (or scan it and e-mail it back to us).

A ppendix B. Confirmation Letter

Confirmation Letter for the AIR Crystal City Office

TO: [INSERT NAME]

JOB#: XXXXXXX

Thank you for accepting our invitation to participate in a research interview. We will begin promptly at the scheduled time and it is important that you arrive 10 minutes early to check in. If you are late, we cannot guarantee that you will be interviewed or paid.

LOCATION:

American Institutes for Research

1400 Crystal Drive, 10th Floor

Arlington, VA 22202

DATE:

TIME:

DURATION: 1 hour

TOPIC: Cognitive Laboratory Interviews for TIMSS 2023

GIFT CARD: Your child will receive a $50 gift card as a thank-you for their participation. If the parent or legal guardian brings the child to and from the interview, they will also receive a $25 gift card as a thank-you.

PLEASE BRING THE FOLLOWING ITEMS WITH YOU TO THE RESEARCH INTERVIEW:

Parent’s or guardian’s driver’s license or valid photo ID
Your child’s glasses, if needed for reading on paper
Signed parental consent form
(8th grade only): Calculator

We look forward to seeing you and would like to stress the importance of your participation in this study. If you find for any reason that you cannot fulfill your commitment, DO NOT invite anyone else to take your place. Instead, please call EurekaFacts at (240) 403-4800 so we can find a replacement and meet our obligation to our clients.

Directions to: American Institutes for Research

1400 Crystal Drive

Arlington, VA 22202

AIR is accessible by car or Metro.

DRIVING DIRECTIONS

FROM MARYLAND – Baltimore-Washington Pkwy/BWI /I-95/CAPITAL BELTWAY

Make your way to the Baltimore-Washington Pkwy
U se the right 2 lanes to take exit
1B-C toward I-695/Route I-395/
Downtown
Continue onto I-695 W (signs for
I-395/Downtown)
Merge onto I-395 S (entering Virginia)
Use the left lane to take exit 8C for US-1 S toward Pentagon City/Crystal City/Alexandria
Take the 15th St S exit toward Pentagon City
Continue on 15th St S to Crystal Dr
Use the left 2 lanes to turn left onto 15th St S
Turn left at the traffic light onto Crystal Dr
Your destination will be the glass building on the left
Proceed to the parking lot below the building
See the next page for parking information

FROM VIRGINIA – I-395 N/I-95/CAPITAL BELTWAY

Make your way to I-395 North
Take exit 8C toward US-1 S/Pentagon City/Crystal City/Reagan National Airport
Turn left onto Army Navy Dr
Turn left onto 12th St S
Continue onto Crystal Dr
Your destination will be the glass building on the right
Proceed to the parking lot below the building
See below for parking information

METRO DIRECTIONS

The nearest Metro stop is Crystal City on the Blue/Yellow Line, a 10-minute walk from the building
After exiting the Metro, turn right (north) and follow the walkway to cross 15th St
Turn right and walk along 15th St
Turn left onto Crystal Dr
Your destination will be the glass building on your left
See below for information on entrance to the building

Parking and entrance to the building:

Participants who will need parking validation should park in the LAZ parking garage below 1400 Crystal Drive. Enter the parking garage and take a ticket at the gate. Park on the G-1 level in any spot marked “RESERVED 24/7 AIR” or in any unmarked parking spot on the G-2 or G-3 level. Head toward the elevators for 1400 Crystal Drive. Please note that you will not be able to access the 10th floor without a key card, so take the elevator to the 1st floor lobby where your interviewer will meet you at the reception desk and will assist you in accessing the 10th floor.

Map and walking directions from Metro:

Please bring your parking ticket with you to the interview. We will reimburse you for your parking costs with presentation of your parking ticket.

Confirmation Letter for the AIR Rockville Office

TO: [INSERT NAME]

JOB#: XXXXXXX

LOCATION:

American Institutes for Research

6003 Executive Blvd, 3rdFloor

Rockville, MD 20852-3823

DATE:

TIME:

DURATION: 1 hour

TOPIC: Cognitive Laboratory Interviews for TIMSS 2023

PLEASE BRING THE FOLLOWING ITEMS WITH YOU TO THE RESEARCH INTERVIEW:

Parent’s or guardian’s driver’s license or valid photo ID
Your child’s glasses, if needed for reading on paper
Signed parental consent form
(8th grade only): Calculator

Directions to: American Institutes for Research (AIR)

6003 Executive Blvd.

Rockville, MD 20852-3823

AIR is accessible by car or Metro.

DRIVING DIRECTIONS

FROM MARYLAND – BWI/ I-95/CAPITAL BELTWAY (MARYLAND)

Keep left to take I-495 W/Capital Beltway W toward Silver Spring
Merge onto I-270 N via EXIT 35 toward Frederick
Merge onto Old Georgetown Rd/MD-187 N via EXIT 1A
Turn left onto Executive Blvd
At the first traffic light, turn right into the parking lot of a three-building campus
Proceed to the parking attendant’s booth to collect a parking ticket for entry
6003 Executive Boulevard North is the building on the far right (east side) as you enter

FROM VIRGINIA – I-95/CAPITAL BELTWAY

Take Exit 170A to I-495 N/Tysons Corner
Cross the Potomac River on the American Legion Bridge
Keep left to take I-270 Spur N via EXIT 38 toward Rockville/Frederick
Take the Democracy Blvd E exit, EXIT 1, bearing right at the fork toward MD-187/Old Georgetown Rd to merge onto Democracy Blvd
Turn left onto Old Georgetown Rd/MD-187
Turn left onto Executive Blvd
At the first traffic light, turn right into parking lot of a three-building campus
Proceed to the parking attendant’s booth to collect a parking ticket for entry
6003 Executive Boulevard North is the building on the far right (east side) as you enter

METRO DIRECTIONS

Closest Metro stop is White Flint on the Red Line, which is a 10-minute walk from the building
Upon exiting the station, turn to your right and take the walkway under Rockville Pike. Continue on Marinelli Road and turn right on Executive Blvd
Cross Old Georgetown Road and 6003 will be the first building on your right

Parking:

There are open lots and a covered garage for parking. Please bring your parking ticket with you to the interview. We will reimburse you for your parking costs with presentation of your parking ticket.

A doorman will greet you, and you will be issued a visitor's pass after presenting identification. Your interviewer will meet you at the main lobby, where you will then be brought to the third floor of the AIR suite.

Appendix C. Parental Consent Document

PARENTAL CONSENT DOCUMENT

FOR PARTICIPATION IN RESEARCH

Title of Research Study: Cognitive Interviews in Support of the Trends in International Mathematics and Science Study (TIMSS) 2023

Thank you for your interest in participating in the research project described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. Contact information is included under question 8, below. If you do not have questions now, you may ask them if they occur to you later. The collection of information in this study is authorized by 20 U.S. Code, Section 9543. The U.S. Office of Management and Budget has approved the data collection under the OMB control number XXXX-XXXX (expires XX/XX/XXXX).

What is this study about?

The National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the U.S. Department of Education, in collaboration with the American Institutes for Research, is conducting interviews with students who are currently in or who have recently completed 4th or 8th grade about new test questions in an assessment called the Trends in International Mathematics and Science Study (TIMSS). Feedback about the questions will help NCES improve the questions and make sure they can be easily understood by students across the country.

For more information on TIMSS 2023, please view the TIMSS brochure at https://timssandpirls.bc.edu/timss2023/downloads/T2023_TIMSS_Brochure.pdf.

Where will the research interviews take place?

The interviews will take place at the American Institutes for Research offices located at either 1400 Crystal Drive, 10th Floor, Arlington, VA 22202, or 6003 Executive Blvd, 3rd Floor, Rockville, MD 20852.

What will happen during the research interview?

Each participant will take a few multi-step math or science test items. An interviewer will ask your child to think aloud while they are completing the test items, and the interviewer will ask a few questions at the end of the interview about your child’s experience and thoughts on the test questions. Your child will not be graded. The purpose of this interview is to gain student feedback on the clarity and difficulty of test questions in order to improve the assessment.

Will the answers be kept confidential and the information private?

Your child will be assigned a unique student identifier, and at no time will your child’s name be linked to any of his/her answers. The interviewers will take notes that will only be used by the researchers. Also, all of your child’s answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

How long will the research interview last and when it will take place?

The interview will last up to 60 minutes and will require only one visit to our office. Interviews will be held Monday through Saturday between 9 am and 8 pm, with the last interview of the day beginning at 7 pm.

Will the results go to my child’s school?

Your child’s answers will not be disclosed to his or her school and will not be linked to any personally identifiable information, such as the school’s name.

What are the possible risks of being in the study?

There are no known risks to participating in this study.

Who can I contact with questions or for further information?

Amy Rathbun is conducting this study at the American Institutes for Research on behalf of NCES. If you have any questions, please call her at (202) 403-6156 or e-mail her at [email protected]. If Amy is not available, please call Frank Fonseca at (202) 403-5612 or e-mail him at [email protected].

Can participation end early?

Your participation in this study is completely voluntary. Your child has the right to refuse to answer particular questions. You may elect to withdraw from this study at any time.

By signing below, you agree that your child, _______________________________, may participate in this study. Your signature below will indicate that your questions have been answered satisfactorily and that you have read and understood the information provided above. Your signature also indicates that you understand that your child’s session will be audio-recorded unless you mark the opt-out box below.

Print Name: ________________________________________

Signature: ___________________________________________________Date:______________________

Shape1

Audio recording opt-out

Appendix D. Thank You E-mail/Letter

Thank You E-mail/Letter to Parents of Student Participants (Under Age 18)

Subject: Thank You for Your Participation – TIMSS Research Study

<DATE>

Dear <NAME>,

On behalf of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences in the U.S. Department of Education, thank you for allowing <CHILD’S NAME> to participate in the Trends in International Mathematics and Science Study (TIMSS) research interview. <CHILD’S NAME>’s input helped us evaluate test questions for the TIMSS assessment. The assistance <he/she> provided, combined with input from other students, was incredibly useful and informative for improving the assessment for students internationally. We greatly appreciate your taking the time to bring <CHILD’S NAME> to and from the interview location to participate.

For more information on TIMSS, please visit the NCES TIMSS website at http://nces.ed.gov/timss/index.asp.

Again, thank you for your assistance.

Sincerely,

Amy Rathbun

American Institutes for Research

(202) 403-6156

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	lmalley
File Modified	0000-00-00
File Created	2021-01-14

Trends in International Mathematics and Science Study (TIMSS 2023) Cognitive Interviews

NCES System Clearance for Cognitive, Pilot, and Field Test Studies 2019-2022

Appendix OMB TIMSS 2023 Cog Labs