Trends in International mathematics and science study (TIMSS 2015) Cognitive Interviews
Appendixes A-f
Screeners
and
Respondent Communication materials
REQUEST FOR OMB Clearance
OMB# 1850-0803 v.86
Submitted by:
National Center for Education Statistics
U.S. Department of Education
Institute of Education Sciences
Washington, DC
September 2013
Table of Contents
A. Recruitment Screener/Phone Script 3
C. Consent Form for Parents of Student Participants (Under Age 18) 13
D. Consent Form for Student Participants (Age 18 or Older) 15
E. Thank You E-mail/Letter to Parents of Student Participants (Under Age 18) 17
F. Thank You E-mail/Letter to Student Participants (Age 18 or Older) 17
Design: 150 total in-depth interviews with 4th, 8th, and 12th graders (50 of each) Specifications:
|
STUDENT’S FIRST NAME:
PARENT’S FIRST NAME (SPOKEN TO / GIVES PERMISSION): _____________________________
DATE RECRUITED: RECRUITED BY:
DATE CONFIRMED: CONFIRMED BY:
ASK TO SPEAK TO THE HEAD OF THE HOUSEHOLD:
Hello, this is from Shugoll Research, an independent market research firm.
We are conducting a brief, but important study about math and science standardized test questions. This is strictly research. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation. I’d like to tell you a little bit about the study first; is now a good time? (YES or NO and call back later)
We are looking for students to help us understand how they interpret test questions. These test questions are part of an international assessment of student achievement which will be conducted in 2015, called the Trends in International Mathematics and Science Study, or TIMSS for short. Participation in this study by the United States at regular intervals provides data on current and past education policies and a comparison of U.S. education policies with its international counterparts. In the U.S., the National Center for Education Statistics (NCES), part of the Department of Education, sponsors this study. The objective of the current study is to assess the clarity of the test questions that will be used for the TIMSS assessment, and we need help from students to do that. The information gained from the students will be used for potential revisions to the test questions. American Institutes for Research (AIR) will be conducting the interviews. I’d like to ask you a few questions to see if you qualify for the study.
P1. Do you have a child in your household who is 18 years old or younger?
|
|
|
|
|
|
|
|
|
P2a. How old is this child?
RECORD AGE
P2b. What grade is your child currently in?
RECORD GRADE
(MUST BE 4th, 8th or 12th GRADE, OTHERWISE, THANK AND TERMINATE)
(PARENTS OF 4TH AND 8TH GRADERS SKIP TO Q.P4, PARENTS OF 12TH GRADERS CONTINUE)
P3. (ASK ONLY TO PARENTS OF 12th GRADERS) Is your child currently enrolled, or has your child taken, a calculus, IB mathematics, or physics class in school?
|
|
|
|
|
|
|
|
|
P4. (ASK TO PARENTS OF 4th AND 8th GRADERS) To what extent do you agree or disagree with each of the following statements? (READ LIST)
|
SELECT ONE PER ROW |
|||||||
|
Agree Strongly |
|
Disagree Strongly |
Don’t Know |
||||
My child received an A or B in math this year, or excelled in math if no letter grade |
7 |
6 |
5 |
4 |
3 |
2 |
1 |
8 |
My child received an A or B in science this year, or excelled in science if no letter grade |
7 |
6 |
5 |
4 |
3 |
2 |
1 |
8 |
|
|
|
|
|
|
|
|
|
(MUST SELECT CODES 6 OR 7 FOR EITHER MATH OR SCIENCE TO CONTINUE. OTHERWISE, THANK AND TERMINATE)
P5. We are conducting a study with students about new math and science standardized test questions. If your student qualifies, do you give your permission for us to invite him/her to participate in a 60-minute interview? The student will receive a $25 gift card for participating, and a parent or legal guardian who brings the student to and from the interview site will also receive a $25 gift card to thank them for their time.
|
SELECT ONE |
|
Yes |
1 |
(CONTINUE) |
|
|
|
No |
2 |
(THANK AND TERMINATE) |
P6. From your perspective, on a scale from 1 to 4 where 1 means “not at all comfortable,” and 4 means “very comfortable” how comfortable would your student be expressing his or her thoughts to a researcher he/she has not met before about how clear and understandable are standardized test questions and instructions presented to him or her? (READ LIST)
|
SELECT ONE |
|
Very Comfortable |
4 |
(CONTINUE) |
Somewhat Comfortable |
3 |
|
|
|
|
Not Too Comfortable |
2 |
(THANK AND TERMINATE) |
Not At All Comfortable |
1 |
P7. If your child participates in the research study, you must sign and return a parental consent form. Your child can either bring the signed form to the interview or you can scan it and email it back. Without a signed form your child cannot participate in the interview and receive the honorarium. May I email you a parental consent form if your child chooses to participate?
|
SELECT ONE |
|
Yes |
1 |
(CONTINUE) |
|
|
|
No |
2 |
(THANK AND TERMINATE) |
RECORD PARENT’S EMAIL ADDRESS
The next few questions are for classification purposes only.
P8. Is your child eligible for free or reduced-price lunch in school? (DO NOT READ LIST)
|
|
|
|
P9. What kind of area do you live in? (READ LIST)
|
|
|
|
|
|||
|
|||
|
|
||
|
P10. Is your child Hispanic or Latino? (DO NOT READ)
|
|
|
|
P11. Which of the following categories best describes your child’s ethnic or racial background? (READ LIST)
|
|
|
|
At this point, I’d like to be able to ask your child a few questions, during which you are welcome to stay on the line. Is he/she available to speak with me at this time? (If not, get a re-call time and call back) What is your student’s name? (RECORD NAME)
Again, you are welcome to stay on the line, but please allow your student to speak for him/herself. Thank you.
RECRUITER: Please determine how talkative and articulate the student is. It is critical that we speak with students who are comfortable expressing themselves with someone they are meeting for the first time. Please use your best judgment and recruit only students that are thoughtful and freely express themselves in a manner that is relevant to the topic being discussed.
All recruits MUST be articulate students who can answer open ended questions, and can provide support or explanation for their responses--that is “I like it because...,” “I don’t like it because...”
DO NOT recruit any who have difficulty answering these questions, whose parent answers FOR them, who seem hesitant or shy speaking with you (an adult they do not know), who answer “I don’t know,” or who CANNOT provide an explanation supporting their responses.
SAY TO STUDENT:
Hi (insert target student’s first name). My name is and I have been speaking with your parent for a few minutes.
As I explained to your parent, I work for a research company and we are conducting an important study about math and science standardized test questions. These test questions are part of an international assessment in which the U.S. takes part. We are looking for students to help us understand what they think about these test questions, so we can help to make the test better. Right now, I’d like to talk to you to find out if you are able to participate in the study. There are no “right” or “wrong” answers to any of these questions, and whatever you say is OK. OK?
Great.
ASK THE FOLLOWING QUESTIONS OF THE STUDENT:
C1. What grade level are you currently in? (DO NOT READ LIST)
RECORD GRADE
(MUST BE 4th, 8th or 12th GRADE, OTHERWISE, THANK AND TERMINATE)
C2. What type of school do you currently attend? (READ LIST)
|
SELECT ONE |
|
Public school |
1 |
(RECRUIT A MIX) |
Private school |
2 |
|
|
|
|
Don’t know |
3 |
(THANK AND TERMINATE) |
C3. What is the name of your school? (RECORD)
_________________________________________________________________________
RECRUIT A MIX OF SCHOOLS
(4TH AND 8TH GRADERS SKIP TO Q.C7, 12TH GRADERS CONTINUE)
C4a. (ASK ONLY OF 12TH GRADERS) Which of the following math classes, if any, are you currently enrolled in or have you ever completed at some point during your high school career? (READ LIST BELOW AND SELECT ALL THAT APPLY, THEN ASK NEXT QUESTION)
C4b. What grade (A, B, C, D or F) do you have in each class if you’re currently enrolled, or what grade did you receive in the class if completed in previous years? (READ EACH CLASS CIRCLED AND ASK FOR LETTER GRADE)
|
|
C4a SELECT ALL THAT APPLY |
C4b ENTER LETTER GRADE |
|
|
Calculus |
1 |
|
®(MUST HAVE AN “A” OR “B” IN ALL CLASSES TAKEN IN ORDER TO QUALIFY FOR MATHEMATICS ASSESSMENT) |
|
Multivariate Calculus |
2 |
|
|
|
Differential Calculus |
3 |
|
|
|
AP Calculus AB |
4 |
|
|
|
AP Calculus BC |
5 |
|
|
|
Particular Topics in Calculus |
6 |
|
|
|
IB Mathematics |
7 |
|
|
|
|
|
|
|
|
None |
8 |
®(CONTINUE) |
|
|
|
|
|
|
(DO NOT READ) |
Refused |
9 |
®(THANK & TERMINATE) |
|
NOTE: OF 25 TOTAL 12TH GRADE RECRUITS, NEED HALF TO QUALIFY FOR MATH ASSESSMENT AND HALF FOR THE PHYSICS ASSESSMENT.
C5a. (ASK ONLY OF 12TH GRADERS) Which of the following science classes, if any, are you currently enrolled in or have you ever completed at some point during your high school career? (READ LIST BELOW AND SELECT ALL THAT APPLY, THEN ASK NEXT QUESTION)
C5b. What grade (A, B, C, D or F) do you have in each class if you’re currently enrolled, or what grade did you receive in the class if completed in previous years? (READ EACH CLASS CIRCLED AND ASK FOR LETTER GRADE)
|
|
C5a SELECT ALL THAT APPLY |
C5b ENTER LETTER GRADE |
|
|
Physics – Advanced Studies |
1 |
|
®(MUST HAVE AN “A” OR “B” IN ALL CLASSES TAKEN IN ORDER TO QUALIFY FOR PHYSICS ASSESSMENT, SKIP TO Q.C8) |
|
AP Physics B |
2 |
|
|
|
AP Physics C |
3 |
|
|
|
IB Physics |
4 |
|
|
|
|
|
|
|
|
None |
8 |
®(CONTINUE IF QUALIFIED FOR MATH ASSESSMENT, OTHERWISE THANK & TERMINATE) |
|
|
|
|
|
|
(DO NOT READ) |
Refused |
9 |
®(THANK & TERMINATE) |
|
NOTE: OF 25 TOTAL 12TH GRADE RECRUITS, NEED HALF TO QUALIFY FOR MATH ASSESSMENT AND HALF FOR THE PHYSICS ASSESSMENT.
C6. (ASK ONLY OF 12TH GRADERS) Do you have a graphing calculator, that is, a calculator you would use for math tests at school?
|
SELECT ONE |
|
Yes |
1 |
(CONTINUE) |
|
|
|
No |
2 |
(THANK & TERMINATE) |
C7. (ASK ONLY OF 4TH AND 8th GRADERS) How well would you say you typically do in your math classes? (READ LIST)
|
|
SELECT ONE |
|
|
Outstanding (letter grade A) |
1 |
(CONTINUE; STUDENT QUALIFIES FOR MATH ASSESSMENT) |
|
Very good (letter grade B) |
2 |
|
|
|
|
|
|
Satisfactory (letter grade C) |
3 |
(CONTINUE) |
|
Needs improvement (letter grade D) |
4 |
|
|
Not achieving (letter grade F) |
5 |
|
(DO NOT READ) |
Refused |
6 |
C8. (ASK ONLY OF 4TH AND 8th GRADERS) How well would you say you typically do in your science classes? (READ LIST)
|
|
SELECT ONE |
|
|
Outstanding (letter grade A) |
1 |
(CONTINUE; STUDENT QUALIFIES FOR SCIENCE ASSESSMENT) |
|
Very good (letter grade B) |
2 |
|
|
|
|
|
|
Satisfactory (letter grade C) |
3 |
(THANK AND TERMINATE IF STUDENT REPORTS BETWEEN 3 AND 6 FOR BOTH C7 and C8;OTHERWISE CONTINUE) |
|
Needs improvement (letter grade D) |
4 |
|
|
Not achieving (letter grade F) |
5 |
|
(DO NOT READ) |
Refused |
6 |
NOTE: OF TOTAL 4th and 8th GRADE RECRUITS, NEED HALF OF EACH GRADE TO QUALIFY FOR MATH ASSESSMENT AND HALF FOR THE PHYSICS ASSESSMENT.
C9. (ASK TO ALL) How comfortable do you think you would feel talking with a researcher you haven’t met before about how clear and understandable are standardized test questions and instructions presented to you? Would you be: (READ LIST)
|
SELECT ONE |
|
Very Comfortable |
4 |
(CONTINUE) |
Somewhat Comfortable |
3 |
|
|
|
|
Not Too Comfortable |
2 |
(THANK AND TERMINATE) |
Not At All Comfortable |
1 |
C10. Please tell me what your favorite subject is in school and why. (WRITE VERBATIM)
|
|
|
|
|
|
|
|
|
|
|
|
NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE. |
|
|
C11. RECORD GENDER OF STUDENT. DO NOT ASK.
|
|
|
|
|
(RECRUIT A MIX) |
||
|
You qualify to participate in an interview session with a researcher in (INSERT LOCATION) to discuss new standardized test questions for (INSERT “MATH” OR “SCIENCE”). The interview is scheduled for (INSERT DAY/DATE AND TIME). The session will last 60 minutes. The interview is strictly for research purposes. Let me assure you that no one will attempt to sell or enroll you in anything. A $25 gift card will be given to each participant.
(IF STUDENT IS UNDER 18) Your parent gave permission for you to attend if you are available. (ALL STUDENTS) Will you be able to attend the discussion?
|
SELECT ONE |
|
Yes |
1 |
(CONTINUE) |
|
|
|
No |
2 |
(THANK AND TERMINATE) |
To attend the session and receive your cash honorarium you must bring the signed INSERT “PARENTAL CONSENT” IF UNDER 18 AND “STUDENT CONSENT” IF 18 OR OVER) form with you that we will email to (INSERT “YOUR PARENT” OR “YOU”) or scan it and email it back to us. Do you understand that you must have a completed (INSERT “PARENTAL CONSENT” IF UNDER 18 AND “STUDENT CONSENT” IF 18 OR OVER) form to participate?
|
SELECT ONE |
Yes |
1 |
|
|
No |
2 |
(FOR 12th GRADERS ONLY) In order to participate, please also bring your graphing calculator with you to the session. Thank you.
(FOR 8th GRADERS ONLY) You may bring your own calculator to the session if you like, but it is not required. Thank you.
EMAIL PARENT CONSENT FORM IF UNDER 18 OR STUDENT CONSENT FORM IF 18 OR OVER
TO:
JOB#: XXXXXXX
Thank you for accepting our invitation to participate in a research interview. We will begin promptly at the scheduled time and it is important that you arrive 15 minutes early to check in. If you are late, we cannot guarantee that you will be interviewed or paid.
LOCATION :
DATE:
TIME:
DURATION:
TOPIC:
GRATUITY:
PLEASE BRING THE FOLLOWING ITEMS TO THE RESEARCH INTERVIEW WITH YOU:
Your driver’s license or valid picture ID - it is required for participation.
Your glasses if needed for reading or watching TV.
[12TH-GRADE STUDENTS ONLY] Your graphing calculator – it is required for participation.
[8TH-GRADE STUDENTS ONLY] You may bring a calculator that you like to use for school work, but this is not required.
Signed parent consent form or, if 18 or over, signed student consent form.
We look forward to seeing you and would like to stress the importance of your participation in this study. If you find for any reason you cannot fulfill your commitment, DO NOT invite anyone else to take your place. Instead, please call us at 301-215-7248 (TELEPHONE CALL CENTER) so we can find a replacement and meet our obligation to our clients.
PARENTAL CONSENT DOCUMENT
FOR PARTICIPATION IN RESEARCH
American Institutes for Research
Title of Research Study: Cognitive Laboratory Interviews for TIMSS 2015 and TIMSS Advanced 2015
Thank you for your interest in participating in the research project described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. Contact information is included under question 8, below. If you do not have questions now, you may ask them if they occur to you later. The collection of information in this study is authorized by 20 U.S. Code, Section 9543. The U.S. Office of Management and Budget has approved the data collection under the valid OMB control number 1850-0803 (expires 09/30/2016).
What is this study about?
The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is conducting interviews with students currently in 4th, 8th, or 12th grades about new test questions for the Trends in International Mathematics and Science Study (TIMSS). The questions we are testing focus on mathematics and science at 4th and 8th grade, and advanced mathematics and physics at 12th grade.
For more information on TIMSS, please visit the NCES TIMSS website at: http://nces.ed.gov/timss/index.asp.
Where will the research interviews take place?
The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>, near <NEARBY LANDMARK OR METRO STATION>.
What will happen during the research interview?
Each participant will answer a few sample TIMSS <INSERT “MATH” OR “SCIENCE”> test questions. An interviewer will ask your child whether he or she believes the questions and instructions are clear and understandable. Your child will not be graded.
Will the answers be kept confidential and information private?
Your child will be assigned a unique student identifier (ID), and at no time will his/her name be linked to any of his/her answers. The interviewers will take notes that will only used by the researchers. Also, all of your child’s answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).
How long will the research interview last and when it will take place?
The interview will last up to 60 minutes. The interview requires only one visit to our office.
Will the results go to my child’s school?
Your child’s answers will not be disclosed to his/her school, and will not be linked to any personally identifiable information such as the school’s name.
What are the possible risks of being in the study?
There are no known risks to participating in this study.
Who can I contact with questions or for further information?
<CONTACT NAME> is conducting this study at American Institutes for Research on behalf of NCES. If you have any questions, please call him/her at <CONTACT PHONE> or by email <CONTACT EMAIL>.
Can participation end early?
Your participation in this study is completely voluntary. Your child has the right to refuse to answer particular questions. You may elect to withdraw from this study at any time.
By signing below, you agree that your child, _______________________________, may participate in this study. Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.
Print Name: ________________________________________
Signature:
___________________________________________________Date:______________________
PARTICIPATION CONSENT DOCUMENT
FOR PARTICIPATION IN RESEARCH
American Institutes for Research
Title of Research Study: Cognitive Laboratory Interviews for TIMSS 2015 and TIMSS Advanced 2015
Thank you for your interest in participating in the research project described below. This consent form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. Contact information is included under question 8, below. If you do not have questions now, you may ask them if they occur to you later. The collection of information in this study is authorized by 20 U.S. Code, Section 9543. The U.S. Office of Management and Budget has approved the data collection under the valid OMB control number 1850-0803 (expires 09/30/2016).
What is this study about?
The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is conducting interviews with students currently in 4th, 8th, or 12th grades about new test questions for the Trends in International Mathematics and Science Study (TIMSS). The questions we are testing focus on mathematics and science at 4th and 8th grade, and advanced mathematics and physics at 12th grade.
For more information on TIMSS, please visit the NCES TIMSS website at: http://nces.ed.gov/timss/index.asp.
Where will the research interviews take place?
The interviews will take place at <NAME OF INTERVIEW LOCATION>, located at <LOCATION>, near <NEARBY LANDMARK OR METRO STATION>.
What will happen during the research interview?
Each participant will answer a few sample TIMSS test <INSERT “MATH” OR “SCIENCE”> questions. An interviewer will ask you whether you believe the questions and instructions are clear and understandable. You will not be graded.
Will the answers be kept confidential and information private?
You will be assigned a unique student identifier (ID), and at no time will your name be linked to any of your answers. The interviewers will take notes that will only used by the researchers. Also, all of your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573).
How long will the research interview last and when it will take place?
The interview will last up to 60 minutes and requires only one visit to our office.
Will the results go to my school?
Your answers will not be disclosed to your school, and will not be linked to any personally identifiable information such as the school’s name.
What are the possible risks of being in the study?
There are no known risks to participating in this study.
Who can I contact with questions or for further information?
<CONTACT NAME> is conducting this study at the American Institutes for Research on behalf of NCES. If you have any questions, please call him/her at <CONTACT PHONE> or by email <CONTACT EMAIL>.
Can my participation end early?
Your participation in this study is completely voluntary. You have the right to refuse to answer particular questions. You may elect to withdraw from this study at any time.
By signing below, you agree that you will participate in this research study. Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.
Print Name: _____________________________________________
Signature:
______________________________________________________Date:___________________
Subject: Thank You for Your Participation
<DATE>
Dear <NAME>,
On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, thank you for allowing <CHILD’S NAME> to participate in the Trends in International Mathematics and Science Study (TIMSS) research interview. <CHILD’S NAME>’s input helped us evaluate test questions for the TIMSS assessment. The assistance <he/she> provided, combined with input from other students, was incredibly useful and informative. We greatly appreciate you taking the time to bring <CHILD’S NAME> to and from the interview location to participate.
For more information on TIMSS, please visit the NCES TIMSS website at: http://nces.ed.gov/timss/index.asp.
Again, thank you for your assistance.
Sincerely,
<CONTACT NAME>
American Institutes for Research
<CONTACT PHONE>
SUBJECT: Thank You for Participating
<DATE>
Dear <NAME>;
On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, thank you for participating in a Trends in International Mathematics and Science Study (TIMSS) research interview. Your input helped us evaluate test questions for the TIMSS assessment. Your assistance, combined with input from other students, was incredibly useful and informative.
For more information on TIMSS, please visit the NCES TIMSS website at: http://nces.ed.gov/timss/index.asp.
Again, thank you for your assistance.
Sincerely,
<CONTACT NAME>
American Institutes for Research
<CONTACT PHONE>
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | lmalley |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |