National Assessment of Educational Progress (NAEP) Science Questionnaire Cognitive Interviews 2017

The purpose of the cognitive interviews is to inform the development of the student, teacher, and school administrator questionnaires prior to pilot testing. Below are the probes that students, teachers, and school administrators will be asked during the cognitive interview. Please note that some probes may not be addressed if time does not permit.

Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students, teachers, and school administrators will be administered all items and probes. The duration of each cognitive interview will be 90 minutes.

Instructions and Generic Probes

The interviewer should ask the participant to read and answer the questions, one at a time. Each question will be either one discrete question (multiple choice or free response) or one matrix question (i.e., an item stem with two or more multiple choice sub-items). The respondent is not supposed to read the question aloud, but should work on the question in the same fashion he/she would during a NAEP test administration or survey. If a respondent indicates they do not know a word or phrase, the interviewer can provide/read words for the respondent.

For most items, generic probes for discrete and matrix items will include the following:

Generic Probes for Discrete Items (Multiple Choice and Free Response)

No.	Probe	Instructions for Interviewer
1	Can you tell me, in your own words, what the question is asking?	Ask this probe AFTER the respondent has answered the question.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
4	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all discrete questions.
5	How could we make it easier to answer this question?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.

Generic Probes for Matrix Items

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the question, can you tell me in your own words what this question is asking you to do?	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific option/sub-item.
2	Can you tell me, in your own words, what [option a., b., c., etc.] means to you?	Ask this probe for all options/sub-items. Please note probes 2-6 should be asked together for one option/sub-item before moving on to the next option/sub-item.
3	Did you find any words or part of [option a., b., c., etc.] confusing?  Yes  No	Ask this probe for all options/sub-items.
4	What did you find confusing? What could we do to make [option a., b., c., etc.] less confusing?	Ask both probes only if the respondent answered YES to the previous question.
5	Would you say answering [option a., b., c., etc.] was very easy, easy, difficult, or very difficult?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all options/sub-items.
6	How could we make it easier to answer [option a., b., c., etc.]?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.

In some cases, items require less in-depth examination. Existing items subjected to minor revisions, such as updated/clarified response options, require administration in cognitive interviews largely to verify that the minor change has not made the existing item unclear or confusing. Other existing items may only need to be administered in cog labs to ensure specific terminology is still relevant (e.g., “science specialist”). In these cases, using a reduced set of generic probes can reduce participant burden. For these items, generic probes will be used (listed on the next page).

Generic Probes for Discrete Items (Multiple Choice and Free Response) – Reduced Set

No.	Probe	Instructions for Interviewer
1	Can you tell me, in your own words, what the question is asking?	Ask this probe AFTER the respondent has answered the question.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.

Generic Probes for Matrix Items – Reduced Set

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the question, can you tell me in your own words what this question is asking you to do?	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific option/sub-item.
2	Can you tell me, in your own words, what [option a., b., c., etc.] means to you?	Ask this probe for all options/sub-items. Please note probes 2-6 should be asked together for one option/sub-item before moving on to the next option/sub-item.
3	Did you find any words or part of [option a., b., c., etc.] confusing?  Yes  No	Ask this probe for all options/sub-items.
4	What did you find confusing? What could we do to make [option a., b., c., etc.] less confusing?	Ask both probes only if the respondent answered YES to the previous question.

Student Items – Grades 4, 8, 12 (Cross Grades)

[Grades 4, 8, and 12 Science: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	a) Can you describe, in your own words, what “sample of water” means to you?	Specific	Required
3	b) What type of "waves" did you think about when you were answering this question?	Specific	Required
3	e) Can you describe, in your own words, what "food chain" means to you?	Specific	Required
4	f) Can you describe, in your own words, what “soil quality” means to you?	Specific	Required
5	h) Can you describe, in your own words, what “parental genes” means to you?	Specific	Required
6	j) When answering this question, were you thinking about predicting the season where you live? If the item asked you to predict the season somewhere else, such as Alaska, would you give a different answer?	Specific	Required

[Grades 4, 8, and 12 Science: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	a) & b) Can you give examples of other science projects that you might create with friends?	Specific	Required
3	c) Can you give examples of other types of volunteer work that you might do with scientific researchers?	Specific	Required
4	e) Can you give examples of other types of scientific instruments you might use outside of school?	Specific	Required
5	g) & h) What types of "clubs or programs" did you think about when answering this sub-item?	Specific	Required

[Grades 4, 8, and 12 Science: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	Would your answers to these items change if you were taking an easier or harder science course?	Specific	Required

[Grades 4, 8, and 12 Science: Student #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required

Student Items – Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 Science: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	a) Can you give examples of other living things that you learn about in school?	Specific	Required
3	c) Can you give examples of other rocks and minerals that you learn about in school?	Specific	Required
4	d) Can you describe, in your own words, what "Technology and engineering" means? When you learn about technology and engineering in school, what topics and things do you learn about?	Specific	Required
5	e) Can you give examples of other tools you use at school to look at small things?	Specific	Required
6	f) Can you give examples of other tools you use at school to making measurements?	Specific	Required
7	g) When you learn about electricity in school, what topics and things do you learn about?	Specific	Required

[Grades 8 and 12 Science: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	b), c), and d) Can you describe, in your own words, what "science topics" means to you?	Specific	Required
3	d) Can you describe, in your own words, what "short video clip" means to you? Do you watch science-related videos in classes other than science?	Specific	Required

[Grades 8, and 12 Science: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required

Student Items – Grade 4 Specific

[Grade 4, Science: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	f) Can you describe, in your own words, what "used evidence" means to you?	Specific	Required
3	h) What types of "news articles about science" did you think about when answering this sub-item?	Specific	Required

[Grade 4, Science: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required
2	VH178924: Can you give examples of other living things that you learn about in school?	Specific	Required
3	VH178925: When you learn about electricity in school, what topics and things do you learn about?	Specific	Required
4	VH178927: Can you give examples of other chemicals that you learn about in school?	Specific	Required
5	VH1978928: Can you give examples of other rocks and minerals that you learn about in school?	Specific	Required

[Grade 4, Science: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required
2	When you were answering these items, were you thinking about what you did in the classroom, outside the classroom, or both?	General	Required
3	VH178930: What sort of scientific tools do you use in your science class at school?	Specific	Required
4	VH178935: Can you describe, in your own words, what "short video clip" means to you? Do you watch science-related videos in classes other than science?	Specific	Required
5	VH178944: Can you describe, in your own words, what "science topics" means to you? Can you provide an example?	Specific	Required

Student Items – Grade 8 Specific

[Grade 8, Science: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	c) Can you describe in your own words, what “evaluated the quality of a set of tests” means to you?	Specific	Required
3	e) Can you describe, in your own words, what "mathematical representations" means to you?	Specific	Required
4	h) Can you describe, in your own words, what "multiple sources" means to you? What types of "sources" did you think about when answering this sub-item?	Specific	Required

Student Items – Grade 12 Specific

[Grade 12, Science: Student #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	a) Can you describe, in your own words, what "testable research questions" means to you?	Specific	Required
3	d) Can you describe, in your own words, what "ethical concerns" means to you?	Specific	Required
4	f) What types of "mathematical rules" did you think about when answering this sub-item?	Specific	Required
5	i) Can you describe, in your own words, what “credible information” means to you?	Specific	Required

[Grade 12, Science: Student #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required
2	What sorts of "careers in science" did you think about when you were answering this item?	Specific	Required

NOTE: STUDENTS WHO SAID THEY WERE AT LEAST SOMEWHAT LIKELY TO PURSUE A CAREER IN SCIENCE WILL RECEIVE THE FOLLOWING

ITEM (LISTED ON THE NEXT PAGE).

[Grade 12, Science: Student #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	c) Can you describe, in your own words, what “need to graduate” means to you?
3	e) and g) What sorts of "science-related college programs" did you think about when you were answering these sub-items?	Specific	Required
4	h) What sorts of "scientific fields" did you think about when you were answering this sub-item?
5	j) and k) What sorts of "science-related work" did you think about when you were answering these sub-items?	Specific	Required

Teacher Items – Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Science: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required

[Grades 4 and 8 Science: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 6.	Reduced	Required

[Grades 4 and 8 Science: Teacher #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	Compare these items against the set you just completed (TeaPrac4_A 01_0 [Grade 4]/TeaPrac8_A 01_0 [Grade 8]). Which of these sets of items best describes what you are doing with your students in science class?	Specific	Required
3	a) What type of "natural phenomena" did you think about when you were answering this sub-item?	Specific	Required

[Grades 4 and 8 Science: Teacher #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	There may be teachers who feel they taught the subject poorly, or that their students did not adequately prepare for the test but did well anyway. What can be done to ensure that these teachers feel comfortable responding to this item honestly?	Specific	Required

[Grades 4 and 8 Science: Teacher #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	Are there other technological resources you would add to this list?	General	Required

[Grades 4 and 8 Science: Teacher #6]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	a) What type of "scientific research" did you think about when you were answering this sub-item?	Specific	Required
3	b) What type of "science clubs" did you think about when you were answering this question?	Specific	Required
4	c) What type of "scientific instruments" did you think about when you were answering this question?	Specific	Required

[Grades 4 and 8 Science: Teacher #7]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required

[Grades 4 and 8 Science: Teacher #8]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

[Grades 4 and 8 Science: Teacher #9]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required

[Grades 4 and 8 Science: Teacher #10]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

[Grades 4 and 8 Science: Teacher #11]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	b) What types of writing assignments did you think about when you were answering this sub-item?	Specific	Required

Teacher Items – Grade 4 Specific

[Grade 4, Science: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

[Grade 4, Science: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

Teacher Items – Grade 8 Specific

[Grade 8 Science: Teacher #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

[Grade 8 Science: Teacher #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

School Items – Grades 4, 8, and 12 (Cross Grades)

[Grades 4, 8, and 12 Science: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required

[Grades 4, 8, and 12 Science: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 6.	Reduced	Required
2	g) GRADE 4 only: Do you have a conventional science department for students in grade 4?	Specific	Required

School Items – Grade 4 Specific

[Grades 4 Science: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 6.	General	Required

[Grades 4 Science: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	General	Required
2	a) What type of "Parent volunteered" science club did you think about when you were answering this sub-item?	Specific	Required
3	d) Can you describe, in your own words, what “industries” means to you? Would “companies” be a better wording choice?	Specific	Required

[Grades 4 Science: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required
2	Can you describe, in your own words, what "science specialist" means to you? Is there a different term you would use?	Specific	Required

[Grades 4 Science: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required
2	Can you describe, in your own words, what "science coach" means to you? Is there a different term you would use?	Specific	Required

School Items – Grade 8 Specific

[Grades 8 Science: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 6.	Reduced	Required

[Grades 8 Science: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 5.	Reduced	Required
2	a) What type of "Parent volunteered" science club did you think about when you were answering this sub-item?	Specific	Required
3	d) Can you describe, in your own words, what “industries” means to you? Would “companies” be a better wording choice?	Specific	Required

[Grades 8 Science: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required
2	Can you describe, in your own words, what "science specialist" means to you? Is there a different term you would use?	Specific	Required

[Grades 8 Science: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required
2	Can you describe, in your own words, what "science coach" means to you? Is there a different term you would use?	Specific	Required

[Grades 8 Science: School #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 6.	General	Required
2	i) What safety equipment did you think about when you were answering this sub-item?	Specific	Required

School Items – Grade 12 Specific

[Grades 12 Science: School #1]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 6.	Reduced	Required

[Grades 12 Science: School #2]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for discrete items, see page 4.	General	Required
2	Does your school have clubs that are initiated or run by students?	General	Required
3	a) What type of "Parent volunteered" science club did you think about when you were answering this sub-item?	Specific	Required
4	d) Can you describe, in your own words, what “industries” means to you? Would “companies” be a better wording choice?	Specific	Required

[Grades 12 Science: School #3]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required
2	Can you describe, in your own words, what "science specialist" means to you? Is there a different term you would use?	Specific	Required

[Grades 12 Science: School #4]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for discrete items, see page 6.	Reduced	Required
2	Can you describe, in your own words, what "science coach" means to you? Is there a different term you would use?	Specific	Required

[Grades 12 Science: School #5]

Item-Specific Probes:

No.	Probe	Type of Probe	Required/Conditional
1	For generic reduced probes for matrix items, see page 5.	General	Required
2	h) What safety equipment did you think about when you were answering this sub-item?	Specific	Required

Teacher and School Debriefing Probes

Debriefing probe:

No.

Probe

Type of Probe

Required/Conditional

Ask the following probe at the end of each teacher and school administrator cognitive interview:

Is there anything else about science that you think we should have asked <teachers/principals> that we did not?

General

Required

General Debriefing and Thank You (For all student, teacher, and school administrator participants)

Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the survey questions?

Is there anything that you think could make [this/these] survey question(s) clearer?

Thank participant for his/her time and provide gift card, as appropriate.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Qureshi, Farah
File Modified	0000-00-00
File Created	2021-01-23

National Assessment of Educational Progress (NAEP) Science Questionnaire Cognitive Interviews 2017

NCES Cognitive, Pilot, and Field Test Studies System

Vol II - NAEP Science Questionnaire Cog Labs 2017