Volume II

National Center for Education Statistics

National Assessment of Educational Progress

Volume II

Survey Questionnaire Pretesting Protocols

National Assessment of Educational Progress (NAEP)

2023 NAEP Family Structure Study

OMB#1850-0803 v.274

September 2020

Table of Contents

Paperwork Burden Statement 3

Authorization and Confidentiality Assurance 3

Interviewer Welcome Script and Assent/Consent 4

Student Welcome Script 4

Instructions and Probes 5

Examples of retrospective think aloud probes ..5

Examples of general content related probes 5

Examples of item specific content related probes 6

Examples of usability/interaction probes 6

Examples of transition screen probes 6

General Debriefing and Thank You 6

Distress Protocol 7

Appendix 9

Paperwork Burden Statement

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 60 minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

Authorization and Confidentiality Assurance

The National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1850-0928 APPROVAL EXPIRES 7/31/2023

The purpose of this pretesting activity is to inform the development of the student questionnaires prior to use in the field. The student will first be asked to go through the questionnaire at their own pace. After completing the questionnaire, the interviewer will follow-up by facilitating a retrospective “think aloud” session with the student to better understand the student’s approach to completing the questionnaire. If further information about the item is needed beyond what the student communicated as part of the “think aloud” exercise, additional probes will be asked. The probes may be related to the content, usability/interactions, or transition screens. Below are examples of the types of probes that may be asked. Please note that some probes may not be addressed if the information is already collected as part of the think aloud or if time does not permit.

Please note the wording and appearance of the survey items may be slightly modified prior to cognitive interviews based on expert feedback. Further, given this pretesting effort is meant to be an iterative process, the wording, appearance, and order of the items (see Appendix within Volume II) and probes may also be modified based on early cognitive interview data.

Interviewer Welcome Script and Assent/Consent

The interviewer and the student should share video at the start of the interview. Before proceeding further, a parent or legal guardian of students under 18 must present their ID to the interviewer. Students over the age of 18 must also present their ID to the interviewer. The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. The interviewer should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. The interviewer should use conversational language throughout the interview.

After answering questions and giving further explanation, begin the interview with the first item.

Note: Students will be administered all items and probes, if time permits. The duration of each cognitive interview will be 60 minutes.¹

Student Welcome Script

[INTERVIEWER BEGINS RECORDING]

Hello, my name is <name > and I have <name> with me listening to our interview.

Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit more about what we are going to do today.

I have some questions that students in schools across the country will be asked to answer about their homes and their families. The questions will be part of the National Assessment of Educational Progress, or NAEP for short.

Some of the questions may be difficult to understand or answer. Some of these items also include new tools that students will use to give their answers. Our goal is to make these questions easier to understand and the tools easier to use. This is where you come in.

We want you to help us find out which questions are easy to understand, and which are difficult to understand. To do that, we will ask you to answer all the questions in the questionnaire first. We will record your on-screen actions as you complete the questions. When you finish, we will start back at the beginning and go through the questions again, but this time we will ask you questions along the way. We will also ask you to perform specific tasks and will write down what you do. There are no right or wrong answers, but please try to answer the questions the best you can. Your feedback will help the National Center for Education Statistics, or NCES, improve the questions and tools. In the next hour we will try to get through as many questions as possible. You may choose to end the interview at any time.

We are recording this interview so that researchers can review the recordings later. This recording is only meant to help us summarize what we learn about the questions we are trying to improve. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you consent to proceed with the study and to have the interview recorded? Do you have any questions?

If, for any reason, the participant is no longer interested in participating or does not wish to be recorded, thank the participant for his or her or their time and end the interview.

I am going to share my screen with you now and give you control so that you can begin the questionnaire.

Instructions and Probes

The interviewer should ask the participant to read and follow the prompts and answer the entire set of questions. The respondent is not supposed to read the question aloud but should work on the question in the same fashion he/she/they would during a NAEP test administration or survey. Each screen of the question set will include a discrete multiple-choice question(s), an interactive question (i.e., a question requiring students to click or drag and drop elements from one screen location to another), or a prompting statement (i.e., included to provide context or direction for an upcoming question). The student’s on-screen actions will be recorded as they complete the questions. The interviewer will observe the student as they complete the items and will only intervene if the student expresses that they are unable to finish without help. After the student completes the whole question set, the interviewer will share the recorded video of the student’s actions and ask the student to start from the beginning and retrace their steps while thinking aloud. If necessary, the interviewer may facilitate this retrospective think aloud session with some generic probes. If needed, the interviewer will then administer probing questions about each screen in the interactive question set. Examples of probes for the complete cognitive interview are shown below.

Examples of retrospective think aloud probes

• What was in your head here?

• What were you thinking?

• What was in your mind?

• Explain to me what you thought we were asking here/wanted you to do here.

• Where did you start when you were answering this question?

• Which part of the question/screen caught your attention first?

• How did you figure out you were done or not done with this question?

Examples of general content related probes

• Can you explain to me what the question/statement on the screen is asking/saying? Please try to explain without repeating the wording of the item.

• Were there any words or parts of this question/statement that were confusing?

  • If yes, What did you find confusing? What could we do to make the question/statement less confusing?

• Would you say it was very easy, easy, difficult, or very difficult to answer this question?

  • If difficult or very difficult, How could we make it easier to answer this question?

• How sure are you about your answer to this item?

 Very Unsure      Unsure       Sure        Very Sure 

• Were you comfortable answering this item?

Examples of item specific content related probes

• There is a picture on the screen. Can you describe what you see there?

• Can you tell what kind of house each of these buildings is?

• Can you explain what “home” means to you?

• Does this list of people include everyone you live with?

  • If no, Who is the person you live with that is not on this list?

• Why did you use/not use the “rename the home” feature?

Examples of usability/interaction probes

• What did you think of the questions overall? 

• Tell us about what you liked and what you disliked about interacting with the questions.

• What did you think about [x interaction]?

  • If participant indicates the interaction was confusing, What do you need to make this easier/clearer?

• What did you like about doing two homes at once/one after the other? What did you NOT like?

• Explain how you could tell where you were in the questionnaire.

• How helpful were the directions on the screen?

Examples of transition screen probes

• What did you think this screen was telling you to do?

• What did you expect to happen [think would happen] when you did this?

General Debriefing and Thank You

Before we finish, I’d like to hear any other thoughts you have about the questions we worked on today.

Is there anything else you would like to tell me about working on the questions?

Is there anything that you think could make these questions clearer or easier to answer?

[INTERVIEWER ENDS RECORDING]

Thank participant for his/her/their time and provide gift card, as appropriate.

Distress Protocol

During the cognitive interviews, all interviewers will be trained to identify and respond appropriately to varying levels of distress. All interviewers will be trained to take notice of the respondent’s emotional state and follow the step by step procedures using the two levels of intervention outlined below. These step by step instructions are designed to respond to the elevated emotional states while also providing the respondent the opportunity to continue with the interview or discussion if she or he or they so chooses.

1. LOW TO MODERATE EMOTIONAL STATE: Respondent is showing some signs of emotional distress but there is no perceived or expressed threat to his/her/their own safety or the safety of others. These signs may include:

• Somatic complaints (stomach ache, headache)

• Irritability

• Physical restlessness (tapping feet, shifting around a lot)

• Regressive/self-soothing behavior (thumb sucking, hair sucking, rocking)

• Whining

• Acting out

• Stuttering (sudden onset)

IF ANY OF THESE SIGNS ARE OBSERVED:

Step 1: If respondent is exhibiting any of these signs of distress, the interviewer will ask one of the following questions:

• How are you doing?

• Are you ok?

• Do you have any questions?

If respondent indicates she or he or they are “okay” and does not have any questions, the interviewer will continue with the interview and bring the respondent back into the discussion.

Step 2: If the respondent does not respond or indicates she or he or they are distressed, the interviewer will ask:

• Do you need to take a short break?

• Do you need to pause the interview?

If the respondent answers “no” to these questions: The interviewer will continue with the interview/bring the respondent back into the discussion.

Step 3: If the respondent answers “yes” to these questions: the interviewer will allow the respondent to take a short break. Some participants may need a moment to collect themselves and will be willing to proceed.

Step 4: After the break the interviewer will ask: “Do you want to keep going?” If “yes”, the interviewer will continue with the interview and bring the respondent back into the discussion.

Step 5: If “no”, the interviewer will say, “Thank you for participating. I would like to provide you with some names of resources you can call if you would like to talk to someone.” If the respondent is under 18, the interviewer will then end the interview and e-mail/call his/her/their parent or legal guardian with the contact information from the counseling resource list (see Appendices L and N). If the respondent is 18 or older, the interviewer will then end the interview, provide the contact information from the counseling resource list directly to the student, and then e-mail the student after the interview with the same resource list (Appendix M).

Step 6: The interviewer will note the use of the distress protocol in the data file.

2. ELEVATED EMOTIONAL STATE: The interviewer will assess the type of distress and respond using the behavioral indicators listed for each of the three scenarios: extreme distress without intent to harm self or others, expressed self-harm, or expressed harm to others.

NOTE: The interviewer SHOULD NOT ask the student about whether or not they are thinking about harming themselves or others. Rather, the interviewer should base their evaluation of the students’ emotional state by observing their behavior and listening to anything the student has to say. The interviewer should ask general questions (e.g., “How are you doing?”) if they require additional information about a student’s emotional state.

1. Extreme distress without intent to harm self or others:

• Uncontrollable crying

• Emotional outbursts, including expressions of rage

• Respondent isn’t making sense when she/he/they are talking

• Dissociation (can’t remember time/place)

• Experiencing flashbacks (like responding to a bad dream)

2. Expressions of self-harm:

• Plans to hurt him/herself/themselves

• Plans to kill himself/herself/themselves

• Statements indicating she/he/they might hurt themselves

• Statements indicating she/he/they might kill themselves

• Statements indicating she/he/they have lost a reason to live or that life has no purpose

3. Expressions of harm to others:

• Plans to hurt someone else

• Plans to kill someone else

• Expressed statements that she/he/they might hurt someone else

• Expressed statements that she/he/they might kill someone else

• Expressed statements about planning or thinking about using a firearm or other weapon

If any of the above behavioral indicators are observed: STOP THE INTERVIEW and say: “We are going to stop the interview today. It seems you are upset, and it may be helpful to talk to a trained counselor. I am going to provide your parents or legal guardians with some resources that you can contact. This includes contact information for organizations that you can use if you want to talk about any feelings or emotions you experience.”

[INTERVIEWER ENDS RECORDING]

Appendix

The appendix includes the screen captures of the items/screens students will complete/see on the questionnaire during the interview. These screen captures include multiple variants of items.

Grades 4, 8, and 12 Family Structure: Student #1, variant A

 

Grades 4, 8, and 12 Family Structure: Student #2, variant A

 

Grades 4, 8, and 12 Family Structure: Student #3, variant A

 

Grades 4, 8, and 12 Family Structure: Student #4, variant A

 

Grades 4, 8, and 12 Family Structure: Student #5, variant A

 

Grades 4, 8, and 12 Family Structure: Student #6, variant A

 

Grades 4, 8, and 12 Family Structure: Student #7, variant A

 

Grades 4, 8, and 12 Family Structure: Student #8, variant A

 

Grades 4, 8, and 12 Family Structure: Student #9, variant A

 

Grades 4, 8, and 12 Family Structure: Student #10, variant A

 

Grades 4, 8, and 12 Family Structure: Student #11, variant A

 

Grades 4, 8, and 12 Family Structure: Student #12, variant A

 

Grades 4, 8, and 12 Family Structure: Student #13, variant A

 

Grades 4, 8, and 12 Family Structure: Student #14, variant A

 

Grades 4, 8, and 12 Family Structure: Student #15, variant A

 

Grades 4, 8, and 12 Family Structure: Student #16, variant A

 

Grades 4, 8, and 12 Family Structure: Student #17, variant A

 

Grades 4, 8, and 12 Family Structure: Student #18, variant A

 

Grades 4, 8, and 12 Family Structure: Student #21, variant A

 

Grades 4, 8, and 12 Family Structure: Student #22, variant A

 

Grades 4, 8, and 12 Family Structure: Student #1, variant B

 

Grades 4, 8, and 12 Family Structure: Student #2, variant B

 

Grades 4, 8, and 12 Family Structure: Student #3, variant B

 

Grades 4, 8, and 12 Family Structure: Student #4, variant B

 

Grades 4, 8, and 12 Family Structure: Student #5, variant B

 

Grades 4, 8, and 12 Family Structure: Student #6, variant B

 

Grades 4, 8, and 12 Family Structure: Student #7, variant B

 

Grades 4, 8, and 12 Family Structure: Student #8, variant B

 

Grades 4, 8, and 12 Family Structure: Student #9, variant B

 

Grades 4, 8, and 12 Family Structure: Student #10, variant B

 

Grades 4, 8, and 12 Family Structure: Student #11, variant B

 

Grades 4, 8, and 12 Family Structure: Student #12, variant B

 

Grades 4, 8, and 12 Family Structure: Student #13, variant B

 

Grades 4, 8, and 12 Family Structure: Student #14, variant B

 

Grades 4, 8, and 12 Family Structure: Student #15, variant B

 

Grades 4, 8, and 12 Family Structure: Student #16, variant B

 

Grades 4, 8, and 12 Family Structure: Student #17, variant B

 

Grades 4, 8, and 12 Family Structure: Student #18, variant B

 

Grades 4, 8, and 12 Family Structure: Student #19, variant B

 

Grades 4, 8, and 12 Family Structure: Student #20, variant B

 

Grades 4, 8, and 12 Family Structure: Student #21, variant B

 

Grades 4, 8, and 12 Family Structure: Student #22, variant B

 

Grades 4, 8, and 12 Family Structure: Student #23, variant B

 

Grades 4, 8, and 12 Family Structure: Student #24, variant B

 

1 Please note that the 60 minutes includes time for introductions, completing the interview, debriefing, and/or time for additional questions/feedback from the participants.