National Assessment of Educational Progress (NAEP) Core Background Questions Cognitive Interviews for Students, Teachers, and School Administrators

The appropriate Paperwork Reduction Act and OMB confidentiality texts are indicated below. This information is conveyed in the appropriate consent forms and letters contained in the appendix for Volume I.

Grade 4 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 60 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].

Grade 8 and 12 Student Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 1.5 hours including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

Teacher or School Administrator Cognitive Interviews

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-0803. The time required to complete this information collection is estimated to take no more than 2 hours including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this research protocol, please write to: U.S. Department of Education, Washington, D.C. 20202-4537.

II. Welcome Script and Assent/Consent

The following scripts should not be read verbatim. The interviewer needs to be familiar enough with the script to introduce the think aloud in a conversational manner. Text written in italics is suggested content for the interviewer to be thoroughly familiar with in advance. The interviewer should project a warm and reassuring manner toward the participant to develop a friendly rapport and as such should use conversational language throughout.

Student (17 or younger) Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that students like you in many different schools will be asked to answer about themselves, their education, and their families. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. Please try to answer the questions to the best of your ability, but there are no wrong answers. Your ideas will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for students like you. This interview is being recorded so researchers can review the recordings later. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

Before continuing, ask the student if he/she has any questions. After answering questions and giving further explanation, continue with an explanation and demonstration of the think aloud process. If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Student (18 or older) Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

Before continuing, ask the student if he/she has any questions. After answering questions and giving further explanation, continue:

Before we begin, I’d like you to read the following consent form to confirm we have your permission to ask you the questions and write down and record your answers.

Give the participant a copy of the student consent form (see Volume I Appendix B). Ask the participant if he/she has any questions. After answering questions and giving further explanation, ask the participant to sign the consent form. Collect the signed consent form from the participant. Then continue with an explanation and demonstration of the think aloud process. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

Teacher Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that teachers like you in many different schools will be asked to answer about themselves, their instructional practices, and their schools. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. Please try to answer the questions to the best of your ability, but there are no wrong answers. Your ideas will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for teachers like you. This interview is being recorded so researchers can review the recordings later. Your information is confidential and the only people who will see anything related to your interview have signed confidentiality agreements.

Before continuing, ask the participant if he/she has any questions. After answering questions and giving further explanation, continue:

Before we begin, I’d like you to read the following consent form to confirm we have your permission to ask you the questions and write down and record your answers.

Give the participant a copy of the consent form (see Volume I Appendix C). Ask the participant if he/she has any questions. After answering questions and giving further explanation, ask the participant to sign the consent form. Collect the signed consent form from the participant. Then continue with an explanation and demonstration of the think aloud process. If the participant is no longer interested in participating, thank the participant for his/her time and end the interview.

School Administrator Participant Script

Hello, my name is <name >.

Thank you for coming here today to help us.

I have some questions that school administrators like you in many different schools will be asked to answer about their school policies and characteristics. The questions will be part of the National Assessment of Educational Progress (NAEP). Some of the questions may not be easy to understand or answer. Please try to answer the questions to the best of your ability, but there are no wrong answers. Your ideas will help the National Center for Education Statistics (NCES), part of the U.S. Department of Education, make these questions clearer for school administrators like you. This interview is being recorded so researchers can review the recordings later. Your information is confidential and the only people who will see anything related to your interview have signed confidentiality agreements.

Before continuing, ask the participant if he/she has any questions. After answering questions and giving further explanation, continue:

Before we begin, I’d like you to read the following consent form to confirm we have your permission to ask you the questions and write down and record your answers.

III. Think Aloud Instructions and Practice

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner.

We want to learn what people think about when answering questions. As you answer each question, I’d like you to think aloud. All that means is to just read the question aloud and then say what you’re thinking as you answer the question. Please tell me whatever is going through your mind as you answer the question.

Sometimes it’s helpful to do a short “think aloud” practice. I’m going to ask you a simple question and ask you to think aloud as you decide on your answer. The question is:

“How many different kinds of fruit did you eat yesterday?”

Interviewer: Some participants will be silent after hearing the question. Immediately encourage the participant to say whatever he/she is thinking. You may need to remind the participant that the answer to the question has to be a number.

After you finish a question, or after you’ve gone through all the questions, I’d like to hear any other comments — good or bad— you have about the question and any changes that would make the question easier to understand or answer.

So, do you have any questions before we start?

Interviewer: After answering questions and giving further explanation, begin the interview with the first item.

IV. Generic Probes and Think Aloud Hints for the Interviewer

If the participant is silent for 5 or more seconds, use the following as a guideline for encouraging the participant to read the question aloud and say his/her thoughts, or to help him/her elaborate the responses.

We’re interested in capturing all the participant’s mental processes while answering these questions. Your goal is to have the participant speak aloud all his or her thoughts while answering the question by asking follow-up probes after each item. Several things will ensure that the data collected are as complete as possible.

If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be more descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouth.” Don’t ask questions that lead the participant’s response. You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such as:

What are you thinking now?
Any other thoughts?
Tell me how you came to pick that answer.

Use your best judgment. If a participant exhibits difficulty reading or understanding an item, probe the participant without biasing the response.

Can you tell me in your own words what that question was asking? [comprehension and interpretation probe]
What does the word [term] mean to you as it’s used in this question? [comprehension and interpretation probe]

Use your best judgment. If a participant appears to have guessed or seems less than certain of the answer he/she has, probe the participant.

How much would you say you know about [topic]? [confidence probe]
How sure are you of your answer? [confidence probe]
How did you come up with that answer? [recall strategy and bias probe]
Was this easy or hard to answer?” [comprehension and recall probe]
How easy or difficult is it to remember [topic]? [recall probe]
Did you find the list of answer choices easy or hard? [response probe]
How easy or hard was it to choose an answer from that list of choices? [response probe]

Use your best judgment. If a participant appears uncomfortable with a word or answering a question, probe the participant.

The question uses the word [term]. Does that sound OK to you, or would you choose something different? [sensitivity probe]
In general, how do you feel about this question? [sensitivity probe]

Use your best judgment. After testing an item, using the above generic probes to understand why the question as written is not clear, you may reword the question in a way that the participant will understand based on shared interviewer feedback or previous interview experiences. If this is done, be sure to document precisely the reworded question that was tested and test the item using relevant generic probes.

In addition to generic probes, interviewer protocols will include item-specific probes. Not all items have an item-specific probe. Item-specific probes are provided only if the researchers deemed it important to learn more about a cognitive process or issue that might not be raised naturally from the think aloud process or generic probing. If the issue to be inquired through the item-specific probe was addressed, the item-specific probe does not need to be asked. Use your best judgment.

V. Behavioral Coding Hints for Interviewer

It is important to note when certain behaviors occur that could be evidence of problems with an item. When the following behaviors occur, note the occurrence and when it occurred:

Misreading or rereading a word or words in the instructions, question, or answer choices
Skipping a word or words in the instructions, question, or answer choices
Physical evidence of frustration (e.g., vocal exasperation, pained expression, long pauses)
Misinterpreting skip instruction (e.g., began reading an item that should have been skipped)

VI. Sample Interviewer Checklist

Note: The following two tables are sample checklists that we have developed for interviewers to help them remember to observe and note certain issues and ask follow up questions. While not part of the actual protocol, these tables may be helpful as a separate sheet that interviewers can use to help guide the interview. For each item, interviewers can choose to use these checklists and/or to free response “Participant Think Aloud Comments / Interviewer Notes” pages that will be provided to interviewers, which are represented in this Volume in the space between the item and the item-specific probes beginning on page 9. Both the checklists and “Participant Think Aloud Comments / Interviewer Notes” pages would have space at the top of the page to record Interviewer Name, Interviewer Date, Interview ID, and Item Number.

Issue [Check Yes or No in adjacent columns]	Yes	No	Were follow up probes asked?	Comments Describe observed issue(s)
Difficulty reading or understanding an item
Guessing or less than certain of answer
Uncomfortable with a word or with answering the question

Behavioral Issue	Present	Not Present	Comments Describe observed behavioral issue(s)
Misreading or rereading a word or words in the instructions, question, or answer choices
Skipping a word or word in the instructions, question, or answer choices
Physical evidence of frustration (e.g., vocal exasperation, pained expression, long pauses)
Misinterpreting skip instructions (e.g., began reading an item that should have been skipped)

VII. Debriefing and Thank You

The following scripts should not be read verbatim. You need to be familiar enough with the information to introduce the think aloud in a conversational manner. In addition, only ask questions below that have not already been addressed by the participant.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

If you were not making comments and answering my extra questions, about how long do you think it would take you to fill out this questionnaire? [Only ask of Grade 12 Students, Teachers, and School Administrators]

How did you feel about the questions you were asked?

Do you feel the questions were easy or difficult to understand?

Did you feel the questions were easy or difficult to answer?

[Teachers and School Administrators Only] Did the terminology used in the questionnaire items seem appropriate?

Thank participant for his/her time. Provide gift card and get signature to acknowledge receipt.

VIII. Student Items and Item-Specific Probes

Interviewer: Proceed in the following manner for each of the survey items in this section:

Direct the participant to the item and ask him/her to read the question (and any preceding or following instructions) out loud
Instruct the participant to answer the question and to think aloud while doing so
Record the participant’s comments
Ask generic and item-specific probes, as appropriate, before going on to the next item

Note: Some items are grade specific and will be asked only of students in that grade. The grades associated with the items are identified in brackets (e.g., [Grade 4 only]) preceding the item.

Note: The introductions and questions being read by students are identified in the boxes on this and the remaining pages. Each item to be tested is presented on its own page. Each item is numbered before the grading bracket for referencing in this Volume.

1. [Grades 4, 8, 12]

In the next several questions, you are asked about your home. If you live in more than one home during the school year answer the questions based on the home where you live most of the time. If you live in two or more homes about the same amount of time, answer the questions based on only one home.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Before we get to the first question, tell me what this introduction is telling you.

What does “home” mean to you?

2. [Grade 4 and 8 only]

VE588972

Including yourself, how many people living in your home are 17 years old or younger?

(A) 1

(B) 2

(D) 4

(E) 5 or more

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Did you remember the instruction about living in more than one home?

Do you live in more than one home during the school year?

[If yes] Did you answer the question based on only one home? How did you decide which home to choose?

[For the remainder of items about the home, keep the respondent’s answer in mind to ascertain and probe when necessary if they are remembering the instruction and keeping the same singular reference to a particular home in mind.]

3. [Grade 12 only]

VE588972

Including yourself, how many people living in your home are 17 years old or younger?

(A) None

(B) 1

(D) 3

(E) 4

(F) 5 or more

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Did you remember the instruction about living in more than one home?

Do you live in more than one home during the school year?

[If yes] Did you answer the question based on only one home? How did you decide which home to choose?

4. [Grade 4 and 8 only]

VE589049

How many people living in your home are 18 years old or older?

(A) 1

(B) 2

(D) 4

(E) 5 or more

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

5. [Grade 12 only]

VE589053

Including yourself, how many people living in your home are 18 years old or older?

(A) 1

(B) 2

(D) 4

(E) 5 or more

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

6. [Grades 4, 8, 12]

VE592165

Do the following people live in your home? Fill in ovals for all that apply.

(A) Mother

(B) Stepmother

(D) Other female guardian

(E) Father

(F) Stepfather

(G) Foster father

(H) Other male guardian

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “foster mother” mean to you?

What does “foster father” mean to you?

What does “other female guardian” mean to you?

What does “other male guardian” mean to you?

With a long response list like that one, sometimes respondents forget the question being asked. I’d like to go over the response list more carefully to make sure you answered the question in the way that we hope students like you would answer the question. [Confirm response by asking each item as if they were separate items (e.g., Does your mother live in your home?; Does your stepmother live in your home?; etc.) The purpose is to ascertain two things: (1) whether students are reporting that they have a particular parent or guardian type as opposed to whether that parent or guardian lives with them in their primary home and (2) whether or not the student marked all that apply.

[Based on the aforementioned probe, if the student did not mark all that apply] Why did you not fill in the oval for [unmarked item(s)]?

7. [Grades 4, 8, 12]

VE589077

How many adults living in your home have a job?

(A) None

(B) 1

(D) 3

(E) 4

(F) 5 or more

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “job” mean to you?

How do you know if someone in your home has a job?

Are there people in your home who may have a job or not, but you don’t know?

Some people have a job only sometimes or for part of a year, but not all year. When that happens, how would you decide who to count as having a job when answering this question?

When answering this question, how did you decide who to consider an “adult”?

Is there anyone in your home who is age 18 or older who has a job, but you did not consider that person to be an adult?

8. [Grade 4 only]

VE589085

How many adults living in your home graduated from high school?

(A) None

(B) 1

(D) 3

(E) 4

(F) 5

(G) I don’t know.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “graduated from high school” mean to you?

How do you know if someone in your home has graduated from high school?

Are there people in your home who might have graduated from high school, but you don’t know?

When answering this question, how did you decide who to consider an “adult”?

Is there anyone in your home who is age 18 or older who graduated from high school, but you did not consider that person to be an adult?

9. [Grade 4 only]

VE589089

How many adults living in your home graduated from college?

(A) None

(B) 1

(D) 3

(E) 4

(F) 5

(G) I don’t know.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “graduated from college” mean to you?

How do you know if someone in your home has graduated from college?

Are there people in your home who might have graduated from college, but you don’t know if they did?

Is there anyone in your home who is age 18 or older who graduated from college, but you did not consider that person to be an adult?

10. [Grades 4, 8, 12]

VE592166

Do you have the following items at your home? Fill in ovals for all that apply.

(A) Access to the Internet

(B) Clothes dryer just for your family

(D) More than six rooms (not including bathrooms)

(E) More than one bathroom

(F) A bedroom of your own

(G) Three or more cars, trucks, or other vehicles that your family uses

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

With a long response list like that one, sometimes respondents forget the question being asked. I’d like to go over the response list more carefully to make sure you answered the question in the way that we hope students like you would answer the question. [Confirm response by asking each item as if they were separate items (e.g., Do you have access to the Internet at your home?; Do you have a clothes dryer just for your family at your home?; etc.) The purpose is to ascertain whether or not the student marked all that apply.

[Based on the aforementioned probe, if the student did not mark all that apply] Why did you not fill in the oval for [unmarked item(s)]?

How many rooms are in your house? [Ask student to name them and count out loud after each room is named]

What does “other vehicle” mean to you?

Does your family have any cars, trucks, or other vehicles that they do not use? [If one or more] Did you include them in your total when considering the last response choice?

[For students who marked at least one item] If you did not have any of those items at your home, how would you respond to this question?

11. [Grades 4, 8, 12]

VE589141

Would you find the following items at most of the homes in your neighborhood? Fill in ovals for all that apply.

(A) Access to the Internet

(B) Clothes dryer

(D) More than six rooms (not including bathrooms)

(E) More than one bathroom

(F) Three or more cars, trucks, or other vehicles

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “neighborhood” mean to you?

What does “most of the homes in your neighborhood” mean to you?

How far away is your school from your home?

Would you say that your school is in your neighborhood?

With a long response list like that one, sometimes respondents forget the question being asked. I’d like to go over the response list more carefully to make sure you answered the question in the way that we hope students like you would answer the question. [Confirm response by asking each item as if they were separate items (e.g., Do most homes in your neighborhood have access to the Internet?; Do most homes in your neighborhood have a clothes dryer?; etc.) The purpose is to ascertain whether or not the student marked all that apply.

[Based on the aforementioned probe, if the student did not mark all that apply] Why did you not fill in the oval for [unmarked item(s)]?

How do you know what items are at other homes in your neighborhood?

12. [Grades 4, 8, 12]

VE589499

How long have you lived in the United States?

(A) All of my life

(B) More than 5 years, but not all of my life

(D) Less than 3 years

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

13. [Grades 4, 8, 12]

VE589153

Since the beginning of the school year, have you talked to any of the following people about possible jobs or careers when you are an adult? Fill in ovals for all that apply.

(A) Your mother or female guardian

(B) Your father or male guardian

(D) Family friend

(E) Teacher

(F) Principal

(G) School counselor

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “family friend” mean to you?

[For students who marked (C)] Did you include younger siblings when you chose (C)?

How much did you have to talk to a person about possible jobs or careers before you would count them in your response?

How would you answer this question if you did none of the talking, but only listened to someone (e.g., your teacher, principal, or school counselor) give a speech or presentation about jobs or careers?

[For students who marked at least one item] If you did not talk to any of those people about possible jobs or careers, how would you have responded to this question?

Who else have you talked to about possible jobs or careers that we didn’t include in the list of choices?

14. [Grades 4, 8, 12]

VE589250

Since the beginning of the school year, have you talked to any of the following people about college? Fill in ovals for all that apply.

(A) Your mother or female guardian

(B) Your father or male guardian

(D) Family friend

(E) Teacher

(F) Principal

(G) School counselor

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

[For students who marked (C)] Did you include younger siblings when you chose (C)?

How much did you have to talk to a person about college before you would count them in your response?

How would you answer this question if you did none of the talking, but only listened to someone (e.g., your teacher, principal, or school counselor) give a speech or presentation about college?

[For students who marked at least one item] If you did not talk to any of those people about college, how would you have responded to this question?

Who else have you talked to about college that we didn’t include in the list of choices?

15. [Grade 12 only]

VE589361

During this school year, which of the following have you done? Fill in ovals for all that apply.

(A) Taken the SAT or ACT College Entrance Exams

(B) Submitted the Free Application for Federal Student Aid (FAFSA)

(D) Been accepted to a 2-year college

(E) Applied to a 4-year college

(F) Been accepted to a 4-year college

(G) Applied to a technical training program (such as electrician, beautician, mechanic, computer programmer, etc.)

(H) Been accepted to a technical training program

(I) Talked with a military recruiter

(J) Been accepted into the military

(K) Applied for a full-time job

(L) Been interviewed for a full-time job

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

[For students who marked at least one item] If you did not do any of those things, how would you respond to this question?

What does “technical training program” mean to you? Were the examples in parentheses helpful? Were the examples in parentheses necessary?

Would you consider belonging to Junior ROTC or applying for a four-year ROTC scholarship as activities that would fit into choices (I) or (J)?

IX. Teacher Items and Item-Specific Probes

Interviewer: Proceed in the following manner for each of the items that follow:

Direct the participant to the item and ask him/her to read the question out loud.
Instruct the participant to answer the question and to think aloud while doing so.
Record the participant’s comments.
Ask generic and item-specific probes, as appropriate, before going on to the next item.

Note: Some items are grade specific and will be asked only of teachers who teach that grade. Grade-specific details are in brackets preceding the item.

16. [Grades 4, 8]

VE577729

Excluding student teaching, how many years have you worked as an elementary or secondary teacher, counting this year?

(A) Less than 1 year

(B) 1–2 years

(D) 6–10 years

(E) 11–20 years

(F) 21 or more years

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

When calculating the number of years, did you or would you include any part-time teaching assignments or long-term substitute assignments?

17. [Grade 8 only]

VE588202

Excluding student teaching, how many years have you taught [insert subject(s)] in grades 6 through 12, counting this year?

(A) Less than 1 year

(B) 1–2 years

(D) 6–10 years

(E) 11–20 years

(F) 21 or more years

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

When calculating the number of years, did you or would you include any part-time teaching assignments or long-term substitute assignments?

18. [Grades 4, 8]

VE577841

Have you been awarded tenure by the school where you currently teach?

(A) Yes

(B) No

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does the term “tenure” mean to you and how is it defined at your school?

19. [Grades 4, 8]

VE577753

Do you hold a regular or standard certificate that is valid in the state in which you are currently teaching?

(A) Yes, I hold a permanent certificate. → Go to Question VE577758.^¹

(B) Yes, I hold a temporary certificate. (This type of certificate may require additional coursework, student teaching, etc.) → Go to Question VE577758.

(D) No, and I am not planning to obtain certification. → Skip to Question VE577783.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Was the difference between a permanent and a temporary certificate clear?

Was the parenthetical description of a temporary certificate helpful?

Was the parenthetical description of a temporary certificate necessary?

If the parenthetical description of a temporary certificate was not included, how would you define a temporary certificate in your own words?

[Behavior code any misinterpretation of skip instructions based on the next question read by the participant.]

20. [Grades 4, 8]

VE577758

Are you certified to teach in the subjects you are currently teaching?

(A) I am certified to teach all subjects that I am currently teaching.

(B) I am certified to teach at least one subject that I am currently teaching.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

21. [Grades 4, 8]

VE592138

Are you certified to teach at the grade levels that you are currently teaching?

(A) I am certified to teach at all grade levels that I am currently teaching.

(B) I am certified to teach at one or more of the grade levels that I am currently teaching.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

22. [Grades 4, 8]

VE577760

Did you enter teaching through an alternative certification program? (An alternative program is a program that was designed to expedite the transition of nonteachers to a teaching career, for example, a state, district, or university alternative certification program.)

(A) Yes → Go to Question VE577783.

(B) No → Skip to Question VE577800.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Was the parenthetical definition of an alternative certification helpful?

Was the parenthetical definition of an alternative certification necessary?

If the parenthetical definition of an alternative certification was not there, how would you define an alternative certification in your own words?

[Behavior code any misinterpretation of skip instructions based on the next question read by the participant.]

23. [Grades 4, 8]

VE577783

Which of the following best describes your occupation immediately prior to becoming a teacher? Fill in only one oval.

(A) College/university student

(B) College/university faculty

(D) Military

(E) Health care professional

(F) Government

(G) Nongovernment, nonprofit organization

(H) Homemaker/caregiver

(I) Retired

(J) Unemployed

(K) Other (specify:)_______________________________________________

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Was filling in only one oval sufficient or would you have preferred the option of selecting more than one response choice?

If more than one option applied, how did you decide which option to choose?

24. [Grades 4, 8]

VE577800

Are you certified by the National Board for Professional Teaching Standards?

(A) Yes, and I am certified in a specific content and/or developmental area. → Go to Question VE592142.

(B) Yes, and I have generalist certification. → Skip to Question VE577837.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Are you familiar with the National Board for Professional Teaching Standards?

Would a parenthetical definition of the National Board for Professional Teaching Standards be necessary?

Would a parenthetical definition of the National Board for Professional Teaching Standards be helpful?

[Behavior code any misinterpretation of skip instructions based on the next question read by the participant.]

25. [Grades 4, 8]

VE592142

In which of the following content areas are you certified by the National Board for Professional Teaching Standards? Fill in ovals for all that apply.

(A) Art

(B) Career and Technical Education

(D) English Language Arts

(E) Exceptional Needs Specialist

(F) Generalist

(G) Health Education

(H) Library Media

(I) Literacy: Reading-Language Arts

(J) Mathematics

(K) Music

(L) Physical Education

(M) School Counseling

(N) Science

(O) Social Studies-History

(P) World Languages Other than English

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

26. [Grades 4, 8]

VE577837

Are you certified in the content areas that you are currently teaching by the National Board for Professional Teaching Standards?

(A) I am certified in all content areas that I am currently teaching.

(B) I am certified in at least one content area that I am currently teaching.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

27. [Grades 4, 8]

VE592146

Are you certified in the developmental areas that you are currently teaching by the National Board for Professional Teaching Standards?

(A) I am certified in all developmental areas that I am currently teaching.

(B) I am certified in at least one developmental area that I am currently teaching.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

28. [Grades 4, 8]

VE577839

Since completing your undergraduate degree, have you taken any graduate courses?

(A) Yes

(B) No

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

Did you take any graduate course as an undergraduate or prior to completing your undergraduate degree? [If yes] Did you count a graduate course when answering the question?

Did you enroll in graduate school or only take graduate courses? [If yes] Were any of these graduate courses necessary for your current teaching position?

X. School Administrator Items and Item-Specific Probes

Interviewer: Proceed in the following manner for each of the items that follow:

Direct the participant to the item and ask him/her to read the question out loud.
Instruct the participant to answer the question and to think aloud while doing so.
Record the participant’s comments.
Ask generic and item-specific probes, as appropriate, before going on to the next item.

Note: Some items are grade-specific and will be asked only of school administrators at schools that serve students in that grade. Grade-specific details are in brackets preceding the item.

29. [Grades 4, 8, 12]

E592238

Can your school be described by any of the following? Fill in ovals for all that apply.

(A) Elementary school

(B) Middle or junior high school

(D) Regular school with a magnet program

(E) A magnet school or a school with a special program emphasis, e.g., science/mathematics school, performing arts school, talented/gifted school, foreign language immersion school

(F) Special education school: primarily serves students with disabilities

(G) Alternative school: offers a curriculum designed to provide alternative or nontraditional education, not clearly categorized as regular, special, or vocational education

(H) Private independent school

(I) Private religiously affiliated school

(J) Independent charter school

(K) Charter school administered by local school district

(L) Other (specify): _____________________

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

30. [Grades 4, 8, 12]

VE592500

As principal, rank the following areas of your focus in order of importance to your school (1= most important to 7= least important).

a. Supporting teacher professional development 

b. Providing ample time (at least one class period) for teacher preparation and consultation with 

other teachers

c. Aligning your beliefs and actions with school goals 

d. Managing conflict 

e. Involving teachers in curriculum development 

f. Involving teacher in hiring decisions 

g. Outreach to parents/community 

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How easy or difficult was it to rank these items?

Would you have preferred to have rated the items on a scale of 1–4 from very important to not important?

If we had provided a rating scale of 1–4 from very important to not important, how could we avoid having principals rate all of these areas as very important?

When you are asked questions like this on confidential surveys, do you find it at all difficult to provide an accurate response? [If yes] why?

31. [Grades 4, 8, 12]

VE592545

As principal, rank the relative amount of time you devoted to each of the following activities during this school year (1= most time to 7 = least time).

a. Administrative duties (e.g., hiring, budgeting, scheduling, meetings) 

b. Instructional leadership (e.g., developing curriculum and pedagogy, planning and supporting 

professional development for teachers)

c. Supervising and evaluating teachers and other staff 

d. Teaching 

e. Public relations and fundraising 

f. Communicating directly with students 

g. Addressing parent/community issues 

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How easy or difficult was it to rank these items?

Would you have preferred to have rated the items on a scale of 1–4 from a great deal of time to not much time?

If we had provided a rating scale of 1–4 from a great deal of time to not much time, how could we avoid having principals rate every activity as a great deal of time?

When you are asked questions like this on confidential surveys, do you find it at all difficult to provide an accurate response? [If yes] why?

32. [Grades 4, 8, 12]

VE588048

Please indicate how much you disagree or agree with each of the following statements about your school. Fill in one oval in each row.

	Strongly disagree	Disagree	Agree	Strongly agree
a. Most teachers are happy to be teaching in this school.	(A)	(B)	(C)	(D)
b. Most teachers have a good understanding of the school’s curricular goals.	(A)	(B)	(C)	(D)
c. Most teachers are successful in implementing the school’s curriculum.	(A)	(B)	(C)	(D)
d. Most teachers have high expectations for student achievement.	(A)	(B)	(C)	(D)
e. Most parents have high expectations for student achievement.	(A)	(B)	(C)	(D)
f. Most parents are involved in school activities.	(A)	(B)	(C)	(D)
g. Most students respect other students.	(A)	(B)	(C)	(D)
h. Most students respect teachers.	(A)	(B)	(C)	(D)
i. Most students respect school property.	(A)	(B)	(C)	(D)
j. Most students want to do well in school.	(A)	(B)	(C)	(D)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

33. [Grade 4 only]

VE588126

To the best of your knowledge, how often does each of the following problems occur among fourth-grade students in your school? Fill in one oval in each row.

	Never	Rarely	Monthly	Weekly	Daily
a. Student in-class misbehavior	(A)	(B)	(C)	(D)	(E)
b. Vandalism	(A)	(B)	(C)	(D)	(E)
c. Theft	(A)	(B)	(C)	(D)	(E)
d. Student bullying	(A)	(B)	(C)	(D)	(E)
e. Physical abuse of students	(A)	(B)	(C)	(D)	(E)
f. Student verbal abuse of teachers	(A)	(B)	(C)	(D)	(E)
g. Student physical abuse of teachers	(A)	(B)	(C)	(D)	(E)
h. Student absenteeism	(A)	(B)	(C)	(D)	(E)
i. Drug and alcohol use by students	(A)	(B)	(C)	(D)	(E)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

When thinking about this question, how were you defining “in your school”?

When some problems that occur or originate outside of school (e.g., drug and alcohol use) affect your fourth-grade students in school, how did you consider that in answering this question?

34. [Grade 8 only]

VE592650

To the best of your knowledge, how often does each of the following problems occur among eighth-grade students in your school? Fill in one oval in each row.

	Never	Rarely	Monthly	Weekly	Daily
a. Student in-class misbehavior	(A)	(B)	(C)	(D)	(E)
b. Vandalism	(A)	(B)	(C)	(D)	(E)
c. Theft	(A)	(B)	(C)	(D)	(E)
d. Student bullying	(A)	(B)	(C)	(D)	(E)
e. Physical abuse of students	(A)	(B)	(C)	(D)	(E)
f. Student verbal abuse of teachers	(A)	(B)	(C)	(D)	(E)
g. Student physical abuse of teachers	(A)	(B)	(C)	(D)	(E)
h. Student absenteeism	(A)	(B)	(C)	(D)	(E)
i. Drug and alcohol use by students	(A)	(B)	(C)	(D)	(E)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

When some problems that occur or originate outside of school (e.g., drug and alcohol use) affect your eighth-grade students in school, how did you consider that in answering this question?

35. [Grade 12 only]

VE592668

To the best of your knowledge, how often does each of the following problems occur among twelfth-grade students in your school? Fill in one oval in each row.

	Never	Rarely	Monthly	Weekly	Daily
a. Student in-class misbehavior	(A)	(B)	(C)	(D)	(E)
b. Vandalism	(A)	(B)	(C)	(D)	(E)
c. Theft	(A)	(B)	(C)	(D)	(E)
d. Student bullying	(A)	(B)	(C)	(D)	(E)
e. Physical abuse of students	(A)	(B)	(C)	(D)	(E)
f. Student verbal abuse of teachers	(A)	(B)	(C)	(D)	(E)
g. Student physical abuse of teachers	(A)	(B)	(C)	(D)	(E)
h. Student absenteeism	(A)	(B)	(C)	(D)	(E)
i. Drug and alcohol use by students	(A)	(B)	(C)	(D)	(E)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

When some problems that occur or originate outside of school (e.g., drug and alcohol use) affect your twelfth-grade students in school, how did you consider that in answering this question?

36. [Grade 12 only]

VE588195

Of the students in last year’s graduating class, approximately what percentage is doing each of the following? Fill in one oval in each row.

	0-5%	6-10%	11-25%	26-50%	51-75%	Over 75%	I don’t know.
a. Attending a two-year college	(A)	(B)	(C)	(D)	(E)	(F)	(G)
b. Attending a four-year college	(A)	(B)	(C)	(D)	(E)	(F)	(G)
c. Attending a vocational-technical or business school	(A)	(B)	(C)	(D)	(E)	(F)	(G)
d. Working for pay	(A)	(B)	(C)	(D)	(E)	(F)	(G)
e. Serving in the military (excluding ROTC and military academies)	(A)	(B)	(C)	(D)	(E)	(F)	(G)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What information does your school collect from graduates that would allow you to answer this question?

What information does your school receive from a data collection service (e.g., National Student Clearinghouse or state government) that would allow you to answer this question?

Does “working for pay” mean full-time, part-time, or both to you?

Is the serving in the military parenthetical reference (excluding ROTC and military academies) necessary?

Is the serving in the military parenthetical reference (excluding ROTC and military academies) helpful?

[If not already asked as part of generic probing, for those who did not respond “I don’t know”]

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer to sub-item X [item not responded to with “I don’t know”]?

Approximations are not expected to be precise, but if you had to guess, what would you say, in plus or minus percentage points, is the margin of error between your estimate and the actual percentage?

Interviewer Note: These probes will repeat for the remaining “percentage” questions. Remind and guide respondent to talk aloud their process for answering the question in a way that reduces the need to repeat these probes, but ask them as necessary to obtain the necessary information for each item.

37. [Grade 12 only]

VB485284

Approximately what percentage of students in your twelfth-grade class has taken at least one course in the following programs? Fill in one oval on each line.

	None	1-5%	6-10%	11-25%	26-50%	Over 50%
a. College Board Advanced Placement^® (AP^®) course	(A)	(B)	(C)	(D)	(E)	(F)
b. International Baccalaureate^® (IB) program	(A)	(B)	(C)	(D)	(E)	(F)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

When thinking of the phrase “has taken”, did you include in your calculation students who are currently enrolled in these classes or only those who had completed a course?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

38. [Grade 12 only]

VE589008

Approximately what percentage of your twelfth-grade students are in each of the following instructional programs?

	0%	1-5%	6-10%	11-25%	26-50%	51-75%	76-90%	Over 90%
a. General high school program	(A)	(B)	(C)	(D)	(E)	(F)	(G)	(H)
b. College prep, academic, or specialized academic (such as science, mathematics, fine arts) program	(A)	(B)	(C)	(D)	(E)	(F)	(G)	(H)
c. Vocational, technical, or business program	(A)	(B)	(C)	(D)	(E)	(F)	(G)	(H)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

What does “vocational, technical, or business program” mean to you?

Are there any type of instructional programs that are not covered in these answer choices?

Would you consider the response choices to be mutually exclusive?

Do you feel the response choices should be mutually exclusive? [If yes] Do you have a suggestion regarding how the response choices could be made mutually exclusive?

39. [Grades 4, 8, 12]

VE592604

Does your school offer tenure to teachers?

(A) Yes → Go to Question VE588721.

(B) No → Skip to Question VE592330.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

[Behavior code any misinterpretation of skip instructions based on the next question read by the participant.]

What does the term “tenure” mean to you, and how is it defined at your school?

40. [Grades 4, 8, 12]

VE588721

Of the following categories of teachers who were full-time teachers at your school at the end of the last school year, what percentage stayed on as full-time teachers for this school year? Fill in one oval in each row.

	0-10%	11-25%	26-50%	51-75%	76-90%	Over 90%
a. Teachers who had at least one year of teaching but were not tenured	(A)	(B)	(C)	(D)	(E)	(F)
b. Tenured teachers	(A)	(B)	(C)	(D)	(E)	(F)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

In your calculation for this question, how would you account for a full-time teacher who remained at the school, but transitioned to another position (e.g., reading specialist)? Would you still consider that person to be a full-time teacher in both years when answering this question?

41. [Grades 4, 8, 12]

VE600319

How many first-year teachers were employed at your school at the end of the last year school?

  

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

When counting the number of teachers, did you include first-year teachers who were not full-time (e.g., part-time teachers or long-term substitutes)?

42. [Grades 4, 8, 12]

VE592330

What percentage of first-year teachers at your school at the end of the last school year stayed on as full-time teachers for this school year?

(A) 0–10%

(B) 11–25%

(D) 51–75%

(E) 76–90%

(F) Over 90%

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

43. [Grades 4, 8, 12]

VE588132

Last school year, approximately what percentage of students at your school enrolled after the first day of school?

(A) 0%

(B) 1–3%

(D) 7–10%

(E) 11–20%

(F) Over 20%

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

44. [Grades 4, 8, 12]

VE592581

Last school year, approximately what percentage of students at your school left before the end of the school year?

(A) 0%

(B) 1–3%

(D) 7–10%

(E) 11–20%

(F) Over 20%

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

45. [Grades 4, 8, 12]

VE588470

During a typical week of school, what is the total number of regularly scheduled volunteers working in the school?

(A) 0

(B) 1–5

(D) 11–15

(E) 16–25

(F) More than 25

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “volunteer” mean to you?

46. [Grades 4, 8, 12]

VE592913

Does your school offer nutrition programs to students on a regular basis?

(A) Yes

(B) No

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What do “nutrition programs” mean to you?

What does “regular basis” mean to you in this question?

47. [Grade 12 only]

VE592328

Does your school offer any of the following services to students on a regular basis? Fill in ovals for all that apply.

(A) Career and technical education workshops

(B) Career counseling services or programs

(D) Career days or job fairs

(E) Career or employment readiness workshops

(F) Nutrition Programs

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “regular basis” mean to you in this question?

What does “technical education workshop” mean to you in this question?

48. [Grades 4, 8, 12]

VE588481

Does your school provide social and emotional learning programs designed to increase student social interactions and school engagement?

(A) Yes →Go to Question VE588652.

(B) No →Skip to Question VE588721.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

[Behavior code any misinterpretation of skip instructions based on the next question read by the participant.]

Was the description of social and emotional learning programs necessary?

Was the description of social and emotional learning programs helpful?

49. [Grades 4, 8, 12]

VE588652

To what extent are each of the following topics included in your school’s social and emotional learning program? Fill in one oval in each row.

	Not at all	Small extent	Moderate extent	Large extent
a. Social skills training	(A)	(B)	(C)	(D)
b. Cognitive-behavior modification	(A)	(B)	(C)	(D)
c. Self-management	(A)	(B)	(C)	(D)
d. Conflict-resolution	(A)	(B)	(C)	(D)
e. Problem-solving	(A)	(B)	(C)	(D)
f. Suicide prevention	(A)	(B)	(C)	(D)
g. Drug abuse prevention	(A)	(B)	(C)	(D)
h. Alcohol abuse prevention	(A)	(B)	(C)	(D)
i. Violence prevention	(A)	(B)	(C)	(D)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What do “small extent,” “moderate extent,” and “large extent” mean to you?

Can you provide the cut-points or examples that would help someone understand how you would distinguish between those three response choices?

Considering both your school and other schools, is there a topic not listed among the choices that one might find in a school’s social and emotional learning program?

50. [Grades 4, 8, 12]

VE588677

Approximately what percentage of students in your school have parents or guardians who do each of the following activities? Fill in one oval in each row.

	Not applicable	0-10%	11-25%	26-50%	More than 50%
a. Volunteer regularly to help in the classroom or another part of the school	(A)	(B)	(C)	(D)	(E)
b. Attend teacher–parent conferences	(A)	(B)	(C)	(D)	(E)
c. Attend cultural, sporting, or social events at the school	(A)	(B)	(C)	(D)	(E)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

How did you calculate the percentage?

Did you come up with a specific or estimated value for the numerator and denominator separately when arriving at your answer?

How sure are you of your answer?

XI. Charter School Administrator Items and Item-Specific Probes

Interviewer: Proceed in the following manner for each of the items that follow:

Direct the participant to the item and ask him/her to read the question out loud.
Instruct the participant to answer the question and to think aloud while doing so.
Record the participant’s comments.
Ask generic and item-specific probes, as appropriate, before going on to the next item.

51. [Grades 4, 8, 12]

VE588799

What is the main focus of your school’s mission? Fill in only one oval.

(A) Developing core skills and meeting specified academic standards

(B) Encouraging rigorous academic performance

(D) Implementing a specific learning approach

(E) Following a specific curriculum

(F) Other (specify): _________________________________________________

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

What does “school’s mission” mean to you?

What does “main focus” mean to you?

Was filling in only one oval sufficient, or would you have preferred the option of selecting more than one response choice?

If asked, would you have been able to rank the first five items from 1=most important to your school’s mission to 5=least important to your school’s mission?

Tell me why you chose X instead of some other answer on the list?

How easy or hard was it to choose an answer?

How easy or hard was it to find your answer on the list?

52. [Grades 4, 8, 12]

VE588849

Who granted your school’s current charter?

(A) School district

(B) State board of education (includes state board of regents and District of Columbia State Board of Education)

(D) State charter-granting agency

(E) City or state public charter school board

(F) Other (specify): ________________________

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

53. [Grades 4, 8, 12]

VE588897

Does your school provide a written contract for parents?

(A) Yes, and parents are required to abide by it. →Go to Question VE588978.

(B) Yes, but signing it is voluntary. →Go to Question VE588978.

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

54. [Grades 4, 8, 12]

VE588978

Are the following elements addressed in your charter-parent contract? Fill in one oval in each row.

	Yes	No
a. Dress code	(A)	(B)
b. Home learning environment	(A)	(B)
c. Homework	(A)	(B)
d. Parent–teacher communication	(A)	(B)
e. Parent volunteering	(A)	(B)
f. School discipline policy	(A)	(B)
g. Student attendance	(A)	(B)
h. Student promotion policy	(A)	(B)
i. Other (specify): ______________________________	(A)	(B)

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

55. [Grades 4, 8, 12]

VE600331

Is this school operated by a company or organization that also operates other charter schools?

(A) Yes

(B) No

Participant Think Aloud Comments / Interviewer Notes:

Item-Specific Probes:

If we were interested in identifying the pedagogical or organizational approach that sets your charter school apart from others, how could we best ask that question?

1 Accession numbers (e.g., VE577758) in skip pattern directions are shown here for reference and will be replaced with actual sequence numbers in the cognitive interview instruments.

File Type	application/msword
Author	BlairJ
Last Modified By	Authorised User
File Modified	2011-01-06
File Created	2010-12-13