National Assessment of Educational Progress (NAEP) Science Questionnaire Cognitive Interviews 2017

NCES Cognitive, Pilot, and Field Test Studies System

Vol II - NAEP Science Questionnaire Cog Labs 2017

National Assessment of Educational Progress (NAEP) Science Questionnaire Cognitive Interviews 2017

OMB: 1850-0803

Document [docx]
Download: docx | pdf


National Center for Education Statistics

National Assessment of Educational Progress




Volume II

Items and Probes



National Assessment of Educational Progress (NAEP) Science Questionnaire Cognitive Interviews 2017





OMB# 1850-0803 v.175











October 2016





The purpose of the cognitive interviews is to inform the development of the student, teacher, and school administrator questionnaires prior to pilot testing. Below are the probes that students, teachers, and school administrators will be asked during the cognitive interview. Please note that some probes may not be addressed if time does not permit.

Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students, teachers, and school administrators will be administered all items and probes. The duration of each cognitive interview will be 90 minutes.

Instructions and Generic Probes

The interviewer should ask the participant to read and answer the questions, one at a time. Each question will be either one discrete question (multiple choice or free response) or one matrix question (i.e., an item stem with two or more multiple choice sub-items). The respondent is not supposed to read the question aloud, but should work on the question in the same fashion he/she would during a NAEP test administration or survey. If a respondent indicates they do not know a word or phrase, the interviewer can provide/read words for the respondent.

For most items, generic probes for discrete and matrix items will include the following:

Generic Probes for Discrete Items (Multiple Choice and Free Response)

No.

Probe

Instructions for Interviewer

1

Can you tell me, in your own words, what the question is asking?

Ask this probe AFTER the respondent has answered the question.

2

Were there any words or parts of this question that were confusing? 

 Yes No

Ask this probe for all discrete questions.

3

What did you find confusing? What could we do to make the question less confusing?

Ask both probes only if the respondent answered YES to the previous question.

4

Would you say it was very easy, easy, difficult, or very difficult to answer this question?

 Very Easy Easy Difficult Very Difficult

Ask this probe for all discrete questions.

5

How could we make it easier to answer this question?

Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.



Generic Probes for Matrix Items

No.

Probe

Instructions for Interviewer

1

Look back at the first part of the question, can you tell me in your own words what this question is asking you to do?

Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific option/sub-item.

2

Can you tell me, in your own words, what [option a., b., c., etc.] means to you?

Ask this probe for all options/sub-items. Please note probes 2-6 should be asked together for one option/sub-item before moving on to the next option/sub-item.


3

Did you find any words or part of [option a., b., c., etc.] confusing?

 Yes No

Ask this probe for all options/sub-items.


4

What did you find confusing? What could we do to make [option a., b., c., etc.] less confusing?

Ask both probes only if the respondent answered YES to the previous question.

5

Would you say answering [option a., b., c., etc.] was very easy, easy, difficult, or very difficult?

 Very Easy Easy Difficult Very Difficult

Ask this probe for all options/sub-items.


6

How could we make it easier to answer [option a., b., c., etc.]?

Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.



In some cases, items require less in-depth examination. Existing items subjected to minor revisions, such as updated/clarified response options, require administration in cognitive interviews largely to verify that the minor change has not made the existing item unclear or confusing. Other existing items may only need to be administered in cog labs to ensure specific terminology is still relevant (e.g., “science specialist”). In these cases, using a reduced set of generic probes can reduce participant burden. For these items, generic probes will be used (listed on the next page).



Generic Probes for Discrete Items (Multiple Choice and Free Response) – Reduced Set

No.

Probe

Instructions for Interviewer

1

Can you tell me, in your own words, what the question is asking?

Ask this probe AFTER the respondent has answered the question.


2

Were there any words or parts of this question that were confusing? 

 Yes No

Ask this probe for all discrete questions.

3

What did you find confusing? What could we do to make the question less confusing?

Ask both probes only if the respondent answered YES to the previous question.



Generic Probes for Matrix Items – Reduced Set

No.

Probe

Instructions for Interviewer

1

Look back at the first part of the question, can you tell me in your own words what this question is asking you to do?

Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific option/sub-item.

2

Can you tell me, in your own words, what [option a., b., c., etc.] means to you?

Ask this probe for all options/sub-items. Please note probes 2-6 should be asked together for one option/sub-item before moving on to the next option/sub-item.


3

Did you find any words or part of [option a., b., c., etc.] confusing?

 Yes No

Ask this probe for all options/sub-items.


4

What did you find confusing? What could we do to make [option a., b., c., etc.] less confusing?

Ask both probes only if the respondent answered YES to the previous question.

Student Items – Grades 4, 8, 12 (Cross Grades)

[Grades 4, 8, and 12 Science: Student #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

a) Can you describe, in your own words, what “sample of water” means to you?

Specific

Required

3

b) What type of "waves" did you think about when you were answering this question?

Specific

Required

3

e) Can you describe, in your own words, what "food chain" means to you?

Specific

Required

4

f) Can you describe, in your own words, what “soil quality” means to you?

Specific

Required

5

h) Can you describe, in your own words, what “parental genes” means to you?

Specific

Required

6

j) When answering this question, were you thinking about predicting the season where you live? If the item asked you to predict the season somewhere else, such as Alaska, would you give a different answer?

Specific

Required









[Grades 4, 8, and 12 Science: Student #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

a) & b) Can you give examples of other science projects that you might create with friends?

Specific

Required

3

c) Can you give examples of other types of volunteer work that you might do with scientific researchers?

Specific

Required

4

e) Can you give examples of other types of scientific instruments you might use outside of school?

Specific

Required

5

g) & h) What types of "clubs or programs" did you think about when answering this sub-item?

Specific

Required







[Grades 4, 8, and 12 Science: Student #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

Would your answers to these items change if you were taking an easier or harder science course?

Specific

Required









[Grades 4, 8, and 12 Science: Student #4]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required





Student Items – Grades 8 and 12 (Cross Grades)

[Grades 8 and 12 Science: Student #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

a) Can you give examples of other living things that you learn about in school?

Specific

Required

3

c) Can you give examples of other rocks and minerals that you learn about in school?

Specific

Required

4

d) Can you describe, in your own words, what "Technology and engineering" means?

When you learn about technology and engineering in school, what topics and things do you learn about?

Specific

Required

5

e) Can you give examples of other tools you use at school to look at small things?

Specific

Required

6

f) Can you give examples of other tools you use at school to making measurements?

Specific

Required

7

g) When you learn about electricity in school, what topics and things do you learn about?

Specific

Required





[Grades 8 and 12 Science: Student #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

b), c), and d) Can you describe, in your own words, what "science topics" means to you?

Specific

Required

3

d) Can you describe, in your own words, what "short video clip" means to you?

Do you watch science-related videos in classes other than science?

Specific

Required





[Grades 8, and 12 Science: Student #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required







Student Items – Grade 4 Specific

[Grade 4, Science: Student #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

f) Can you describe, in your own words, what "used evidence" means to you?

Specific

Required

3

h) What types of "news articles about science" did you think about when answering this sub-item?

Specific

Required





[Grade 4, Science: Student #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required

2

VH178924: Can you give examples of other living things that you learn about in school?

Specific

Required

3

VH178925: When you learn about electricity in school, what topics and things do you learn about?

Specific

Required

4

VH178927: Can you give examples of other chemicals that you learn about in school?

Specific

Required

5

VH1978928: Can you give examples of other rocks and minerals that you learn about in school?

Specific

Required





[Grade 4, Science: Student #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required

2

When you were answering these items, were you thinking about what you did in the classroom, outside the classroom, or both?

General

Required

3

VH178930: What sort of scientific tools do you use in your science class at school?

Specific

Required

4

VH178935: Can you describe, in your own words, what "short video clip" means to you?

Do you watch science-related videos in classes other than science?

Specific

Required

5

VH178944: Can you describe, in your own words, what "science topics" means to you? Can you provide an example?

Specific

Required









Student Items – Grade 8 Specific

[Grade 8, Science: Student #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

c) Can you describe in your own words, what “evaluated the quality of a set of tests” means to you?

Specific

Required

3

e) Can you describe, in your own words, what "mathematical representations" means to you?

Specific

Required

4

h) Can you describe, in your own words, what "multiple sources" means to you?

What types of "sources" did you think about when answering this sub-item?

Specific

Required







Student Items – Grade 12 Specific

[Grade 12, Science: Student #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

a) Can you describe, in your own words, what "testable research questions" means to you?

Specific

Required

3

d) Can you describe, in your own words, what "ethical concerns" means to you?

Specific

Required

4

f) What types of "mathematical rules" did you think about when answering this sub-item?

Specific

Required

5

i) Can you describe, in your own words, what “credible information” means to you?

Specific

Required







[Grade 12, Science: Student #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required

2

What sorts of "careers in science" did you think about when you were answering this item?

Specific

Required



NOTE: STUDENTS WHO SAID THEY WERE AT LEAST SOMEWHAT LIKELY TO PURSUE A CAREER IN SCIENCE WILL RECEIVE THE FOLLOWING

ITEM (LISTED ON THE NEXT PAGE).





[Grade 12, Science: Student #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

c) Can you describe, in your own words, what “need to graduate” means to you?



3

e) and g) What sorts of "science-related college programs" did you think about when you were answering these sub-items?

Specific

Required

4

h) What sorts of "scientific fields" did you think about when you were answering this sub-item?



5

j) and k) What sorts of "science-related work" did you think about when you were answering these sub-items?

Specific

Required





Teacher Items – Grades 4 and 8 (Cross Grades)

[Grades 4 and 8 Science: Teacher #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required





[Grades 4 and 8 Science: Teacher #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 6.

Reduced

Required





[Grades 4 and 8 Science: Teacher #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

Compare these items against the set you just completed (TeaPrac4_A 01_0 [Grade 4]/TeaPrac8_A 01_0 [Grade 8]). Which of these sets of items best describes what you are doing with your students in science class?

Specific

Required

3

a) What type of "natural phenomena" did you think about when you were answering this sub-item?

Specific

Required





[Grades 4 and 8 Science: Teacher #4]



Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

There may be teachers who feel they taught the subject poorly, or that their students did not adequately prepare for the test but did well anyway. What can be done to ensure that these teachers feel comfortable responding to this item honestly?

Specific

Required





[Grades 4 and 8 Science: Teacher #5]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

Are there other technological resources you would add to this list?

General

Required





[Grades 4 and 8 Science: Teacher #6]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

a) What type of "scientific research" did you think about when you were answering this sub-item?

Specific

Required

3

b) What type of "science clubs" did you think about when you were answering this question?

Specific

Required

4

c) What type of "scientific instruments" did you think about when you were answering this question?

Specific

Required





[Grades 4 and 8 Science: Teacher #7]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required





[Grades 4 and 8 Science: Teacher #8]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





[Grades 4 and 8 Science: Teacher #9]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required





[Grades 4 and 8 Science: Teacher #10]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





[Grades 4 and 8 Science: Teacher #11]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

b) What types of writing assignments did you think about when you were answering this sub-item?

Specific

Required







Teacher Items – Grade 4 Specific

[Grade 4, Science: Teacher #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





[Grade 4, Science: Teacher #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





Teacher Items – Grade 8 Specific

[Grade 8 Science: Teacher #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





[Grade 8 Science: Teacher #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





School Items – Grades 4, 8, and 12 (Cross Grades)

[Grades 4, 8, and 12 Science: School #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required





[Grades 4, 8, and 12 Science: School #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 6.

Reduced

Required

2

g) GRADE 4 only: Do you have a conventional science department for students in grade 4?

Specific

Required





School Items – Grade 4 Specific

[Grades 4 Science: School #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 6.

General

Required





[Grades 4 Science: School #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

General

Required

2

a) What type of "Parent volunteered" science club did you think about when you were answering this sub-item?

Specific

Required

3

d) Can you describe, in your own words, what “industries” means to you? Would “companies” be a better wording choice?

Specific

Required





[Grades 4 Science: School #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required

2

Can you describe, in your own words, what "science specialist" means to you? Is there a different term you would use?

Specific

Required





[Grades 4 Science: School #4]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required

2

Can you describe, in your own words, what "science coach" means to you? Is there a different term you would use?

Specific

Required









School Items – Grade 8 Specific

[Grades 8 Science: School #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 6.

Reduced

Required





[Grades 8 Science: School #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for matrix items, see page 5.

Reduced

Required

2

a) What type of "Parent volunteered" science club did you think about when you were answering this sub-item?

Specific

Required

3

d) Can you describe, in your own words, what “industries” means to you? Would “companies” be a better wording choice?

Specific

Required





[Grades 8 Science: School #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required

2

Can you describe, in your own words, what "science specialist" means to you? Is there a different term you would use?

Specific

Required







[Grades 8 Science: School #4]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required

2

Can you describe, in your own words, what "science coach" means to you? Is there a different term you would use?

Specific

Required





[Grades 8 Science: School #5]






Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 6.

General

Required

2

i) What safety equipment did you think about when you were answering this sub-item?

Specific

Required





School Items – Grade 12 Specific

[Grades 12 Science: School #1]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 6.

Reduced

Required





[Grades 12 Science: School #2]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic probes for discrete items, see page 4.

General

Required

2

Does your school have clubs that are initiated or run by students?

General

Required

3

a) What type of "Parent volunteered" science club did you think about when you were answering this sub-item?

Specific

Required

4

d) Can you describe, in your own words, what “industries” means to you? Would “companies” be a better wording choice?

Specific

Required





[Grades 12 Science: School #3]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required

2

Can you describe, in your own words, what "science specialist" means to you? Is there a different term you would use?

Specific

Required





[Grades 12 Science: School #4]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for discrete items, see page 6.

Reduced

Required

2

Can you describe, in your own words, what "science coach" means to you? Is there a different term you would use?

Specific

Required





[Grades 12 Science: School #5]

Item-Specific Probes:

No.

Probe

Type of Probe

Required/Conditional

1

For generic reduced probes for matrix items, see page 5.

General

Required

2

h) What safety equipment did you think about when you were answering this sub-item?

Specific

Required





Teacher and School Debriefing Probes

Debriefing probe:

No.

Probe

Type of Probe

Required/Conditional

1

Ask the following probe at the end of each teacher and school administrator cognitive interview:

Is there anything else about science that you think we should have asked <teachers/principals> that we did not?

General

Required

General Debriefing and Thank You (For all student, teacher, and school administrator participants)


Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.


Is there anything else you would like to tell me about working on the survey questions?


Is there anything that you think could make [this/these] survey question(s) clearer?



Thank participant for his/her time and provide gift card, as appropriate.



File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorQureshi, Farah
File Modified0000-00-00
File Created2021-01-23

© 2024 OMB.report | Privacy Policy