Trends in International Mathematics and Science Study (TIMSS) 2019 Pretest

Trends in International Mathematics and Science Study (TIMSS) 2019 Pretest

OMB# 1850-0803 v.238

Volume 2

Submitted by:

National Center for Education Statistics (NCES)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

September 2018

PAPERWORK BURDEN STATEMENT

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to be an average of 105 minutes at grade 4 and 125 minutes at grade 8, including the time to review instructions and participate in the information collection.

If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any other concerns, please write to: Trends in International Mathematics and Science Study (TIMSS), National Center for Education Statistics, Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

OMB No. 1850-0803 Approval Expires 07/31/2019

INTRODUCTION TO THE ASSESSMENT

For each of the pretesting sessions described in Volume 1, the administrators will provide the following additional instructions to the students:

For this part of the study, we would like you to take your time exploring the system and interacting with the questions and system tools in different ways. While we would like you to provide answers to questions, we would like you to focus mainly on using the system tools and responding to the questions. Any response is acceptable for the questions as we really want you to experiment with how you can respond to questions and move from question to question. Have a little fun with things and explore the system!

Please take the assessment in this way until your time is up.

STUDENT QUESTIONNAIRE: EXPERIENCE WITH ETIMSS

At the end of the cognitive assessments, students will take a brief questionnaire about their experience with the eTIMSS player. This questionnaire will be administered on the tablets.

Grade 4 eTIMSS Questionnaire:

1. A. Did you like that this test was on a computer or tablet?

 I liked it a lot

 I liked it a little

 I didn’t like it very much

 I didn’t like it at all

B. Did you have any of these difficulties?

Click one circle for each row.

	Yes	No
It was hard to type		
I had trouble using the number pad		
Objects were hard to drag		
There was no good place to work out my answers		
The computer or tablet was slow		
I had to start my test over because of a computer or tablet problem		

2. At school this year, how often did you use a computer or tablet to do each of the following?

Click one circle for each row.

	Every day or almost every day	Once or twice a week	Once or twice a month	Never or almost never
Work on a school assignment such as a paper, report, or presentation				
Mathematics schoolwork				
Science schoolwork				
Take a test or quiz				

3. How much do you agree with these statements?

Click one circle for each row.

	Agree a lot	Agree a little	Disagree a little	Disagree a lot
I am good at using a computer				
I am good at typing				
I can use a touchscreen on a computer, tablet, or smartphone				
It is easy for me to find information on the Internet				
I can look up the meanings of words on the Internet				
I can write sentences and paragraphs using a computer				
I can edit text on a computer				

Grade 8 eTIMSS Questionnaire:

1. A. Did you like that this test was on a computer or tablet?

 I liked it a lot

 I liked it a little

 I didn’t like it very much

 I didn’t like it at all

B. Did you have any of these difficulties?

Click one circle for each row.

	Yes	No
It was hard to type		
I had trouble using the number pad		
Objects were hard to drag		
There was no good place to work out my answers		
The computer or tablet was slow		
I had to start my test over because of a computer or tablet problem		

2. At school this year, how often did you use a computer or tablet to do each of the following?

Click one circle for each row.

	Every day or almost every day	Once or twice a week	Once or twice a month	Never or almost never
Work on a school assignment such as a paper, report, or presentation				
Mathematics schoolwork				
Science schoolwork				
Take a test or quiz				

3. How much do you agree with these statements?

Click one circle for each row.

	Agree a lot	Agree a little	Disagree a little	Disagree a lot
I am good at using a computer				
I am good at typing				
I can use a touchscreen on a computer, tablet, or smartphone				
It is easy for me to find information on the Internet				
I can look up the meanings of words on the Internet				
I can write sentences and paragraphs using a computer				
I can edit text on a computer				

4. How well do you know the meaning of each of the following terms?

Click one circle for each row.

	Very well	Well	A little	Not at all
WiFi				
Firewall				
Instant messaging				
Cut and paste				
Spreadsheet				
Icon				
Drag and drop				
Scroll				

STUDENT DEBRIEFING QUESTIONS

After all students have finished the assessment in each session, field staff will conduct a debriefing activity with the students. Field staff will ask the debriefing questions out loud to the group of students. The questions may be paraphrased, and additional follow-up questions, such as those indicated in the bulleted lists below, may be asked. Students will respond by raising their hands and by calling out responses, as appropriate. Field staff will use their discretion about whether some or all of the questions will be asked, depending on the amount of prompting the students require, and on the amount of time remaining. The debriefing session should not exceed 10 minutes.

Field staff will write down the students’ responses to the questions.

1. What did you like the most about the assessment?

[Follow-up questions may include:]

• Can you explain that more?

• What did you like the least about the assessment?

• What was missing from the assessment?

• If you had to take a test, would you rather take it on a tablet, or on paper? Why?

• What do you like most about taking a test on a tablet? What do you like the least about taking a test on a tablet?

2. Did you see anything you thought was strange while taking the assessment?

[Follow-up questions may include:]

• Can you explain that more?

• Did questions or images seem slow to load at all to you?

• Was the system slow to move from one page/question to the next at all?

• Did you notice any freezes of your tablet or any error messages?

• Were you able to make these errors occur when you wanted to? How?

3. Did you find the system easy to use and/or easy to learn to use?

[Follow-up questions may include:]

• How did you like the tutorial?

• Did you use the Help screens? Were the Help screens useful?

• What did you think of the tools available? Did you use them?

• Which tool or tools did you like the most? Why?

• Did you have any difficulty using any of the tools? What did you find difficult?

• How did you find the mouse, stylus, and keypad worked with the assessment?

• Did you have any difficulty responding to questions using the available controls? If so, what made it difficult?

4. Is there anything you think would make the assessment better? If so, what?

5. Is there anything else you want to tell us about taking assessments on tablets?

10