National Assessment of Educational Progress (NAEP) Socioeconomic Status (SES) Indicator Items Development Studies

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is 1 hour, including the time to review instructions and complete the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

NCES is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622.

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

OMB No. 1850-0803 Approval Expires 07/31/2019

Focus Group Protocol

Part I. Interviewer Welcome Scripts and Assent/Consent

Student Participant Welcome Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the focus groups in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored by the interviewer.

NOTE: Parental/legal guardian consent will be obtained before the focus group begins. Parents/legal guardians will be asked to remain in the lobby/waiting area during the discussion.

[Welcome participants and thank them for participating in this focus group.]

[Introduce yourself and the note taker.]

[If observers are present, acknowledge their presence and state that they will not be participating.]

We work for a company called Westat that is in Rockville, Maryland. At Westat, we help a lot of government agencies design questions for surveys. One of these agencies is the National Center for Education Statistics—or NCES. NCES conducts a survey called the National Assessment of Educational Progress—or NAEP. This is a survey taken every year by students who are in the 4th, 8th, and 12th grades that year [For 8th and 12th graders—You may have already taken it once or twice.] We are currently helping NCES try to improve some questions on NAEP that ask about who students live with in their homes. The questions are being changed so that every student can answer the questions, no matter who they live with. We want to make sure these questions are easy to understand and answer, which is why we’re here with you today.

Before we start, I want to tell you a little more about today’s discussion.

First of all, the most important rule today is that there are no right or wrong answers. Your thoughts are very important to us, and we want you to feel free to tell us exactly what you think. We will never share anything you say with your parents/ legal guardians, your teachers, or anyone outside of our research team.

Since there are no right or wrong answers in our discussion today, it’s ok if you don’t agree with each other about everything we talk about. Also, you don’t need to raise your hand when you want to say something. Just jump in, as long as only one person talks at a time. You don’t need to answer every question I ask, but I do hope you’ll jump in when you have something to say!

My job today is to make sure that each one of you gets to share what you want to say, and sometimes I might ask you extra questions just to make sure I understand what you mean.

The session is being audio-recorded. The microphone is there. (Point to the location of the tape-recorder.) The recording will help us make sure we can write a good report when we go back to our offices. If you do not want to be audio recorded, you do not have to participate in today’s conversation. (Excuse any students that do not give assent to audio recording).
(IF IN FACILITY) You may have noticed this mirror behind me. It’s actually a one-way mirror. One of my co-workers is sitting in a room behind the mirror and is going to observe our conversation. They are here to help me make sure I capture everything you say. At the end of our time, I might go back there to see if they have any other questions they want me to ask you.
Even though we are recording our discussion, your identity will be kept private; this means that others will know what was said but not who said it. Anything you say today might be used in my report, but I would only say that a [4th/8th/12th] grade [boy/girl] in [LOCATION] said this. All study materials will be kept safely in a locked office, and once we are done with our work, they will be destroyed.
To help keep your information private, I would ask you all to please not share what you hear in our discussion with others outside the room once you leave here today.
We will end around [END TIME].
Finally, if at any point you do not understand something that I have said, please stop me and I will do my best to explain.

Does anyone have any questions about why we’re here or what we’ll be doing? [Answer questions.]

(HAND OUT ASSENT INFORMATION BROCHURES, REVIEW THEM WITH YOUTH, ANSWER QUESTIONS, AND GET VERBAL ASSENT.)

So, let’s get started.

Part II. Focus Group Activities and Questions

The purpose of the focus groups is to learn how students in non-traditional households think and talk about the important adults in their households. We also want to learn how much detail students at grades 4, 8 and 12 have about these adults’ educational attainments and what kind of work they do. Below are the activities the students will be asked to participate in and the questions they will be asked during the 60-minute focus group session. Please note that some questions may not be addressed if time does not permit.

Sociogram Activity

Prior to students entering the focus group room, make sure each seat has an 11x17 piece of paper, colored markers (and, for 4th graders, star-shaped stickers). Introduce the activity to students by offering an example of a household composition (e.g., multi-generational household, living with adults other than one’s birth parent(s)). Then ask students to go to one of the drawing stations and do the following:

Draw a box to represent each home in which the student lives either all of the time or some of the time (e.g., students who divide their time between two households should draw two boxes).
In each box, draw an “X” for each adult (over 18) who lives in that household with the student. Note that this could include older siblings if they are 18 years old or older and still live at home.
Next to each “X”, write down (or say) how that adult is related to the student (e.g., “mom,” “stepmom,” “dad’s girlfriend”).
Ask each student to draw a star (or put a star-shaped sticker) next to the “X” for any adult who:

The student thinks of as his/her parent.
The student thinks of as his/her legal guardian.
The student believes brings home money that helps pay for household things, like bills and groceries.
May sign forms for the student for school, sports, or other important activities.
Gives the student advice or helps him/her with school work.

Discussion Questions

Each question will be asked of the group, and students can participate in discussion whenever they choose. This will preserve the voluntary nature of the discussion. Note that much of the focus group discussion will be unscripted and that these questions are intended to get the conversation started; the moderators will foster group interaction as the participants begin to respond to the questions.

How did you decide who to mark as your parent?
How did you decide who to mark as your legal guardian? What do you think is meant by “legal guardian”? How is that different than a parent?
We asked you to mark with a star each adult who brings home money to pay for household things. How did you decide which adults should get a star? How sure are you about your answers? Please explain.
And how did you decide who to mark as the adults who sign forms for you?
And what about the adults who give you advice or help you with school work?
Some of you have adults in your drawings who did not get a star next to them. If you are comfortable doing so, please tell us about your decision not to give them stars.
NON-TWO HOUSEHOLD GROUPS ONLY PROBE: Do any of you spend time in a second household? Tell us why you decided not to draw a second box.

Education Level of These Important Adults

This section of the discussion will explore how much students in each grade level know about each important (starred) adult’s level of educational attainment. Students should be reminded that we are not sure how hard this question is to answer, so they should not feel like they have to know the answer for each (or any) adult in their drawing. The moderator should also put the existing response categories on the easel for students to use if they are unable to offer their parent’s/legal guardian’s education level on their own:

Person did not finish high school.
Person graduated from high school.
Person had some education after high school.
Person graduated from college.
I don’t know.

Let’s look at the individuals with stars in each of your drawings. (GO AROUND THE ROOM.) For each of these starred adults, what can you tell me about how far they went in school?

PROBE: How do you know this?

FOR THOSE WHO SAID “I DON’T KNOW”: What do you know about this person’s education level, if anything?

FOR EACH PARTICIPANT: How easy or difficult is it to answer this education question for each of these adults? Please explain.

Occupation of These Important Adults

This section of the discussion will explore how much students in each grade level know about each important adult’s work life and occupation. Students should be reminded that we are not sure how hard this question is to answer, so they should not feel like they have to know the answer for each (or any) adult in their drawing.

Now let’s talk about work. (GO AROUND THE ROOM.) For each of these starred adults, put a capital “W” next to each of the ones who work.

PROBES: What do you think I mean by the word “work”?

For those with a “W”, what can you tell me about the kind of work they do?

FOR EACH PARTICIPANT: How easy or difficult is it to answer this work question for each of these adults? Please explain.

Closing Remarks and Provision of Incentives

Ask students if they have any additional thoughts or questions. Thank the students and close the session. Remind them to pick up his or her incentive on the way out.

Cognitive Testing Items and Probes

The purpose of the cognitive interviews is to inform the development of the household composition items for the 4th grade, 8th grade, and 12th grade student questionnaires prior to pilot testing. Below are example probes that students will be asked during the cognitive interview. Please note that some probes may not be addressed if time does not permit. The exact question wording of the items and associated probes below will be finalized after the focus groups.

Part I. Interviewer Welcome Script and Assent/Consent

The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with the script to introduce the participant to the cognitive interview process in a conversational manner. The text in italics is suggested content for you to become thoroughly familiar with in advance. You should project a warm and reassuring tone toward the participant in order to develop a friendly rapport. You should use conversational language throughout the interview.

Hello, my name is [NAME].

Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit more about what we are going to do today.

I have some questions that students in many different schools across the country will be asked to answer about themselves and the people they live with in their homes. The questions will be part of the National Assessment of Educational Progress –or NAEP.

Some of the questions may be difficult to understand or answer. Our goal is to make these questions better so that other students can easily understand them. That is why we need your feedback.

We want you to help us find out which questions are easy to understand and which are difficult to understand. There are no right or wrong answers. Your feedback will help the National Center for Education Statistics—or NCES—which is part of the United States Department of Education, make these questions clearer for students like you. In the next hour, we will try to get through as many questions as possible.

IF RESPONDENT HAS AGREED TO RECORDING: This interview is being audio recorded so that researchers can review the recordings later. No one else will hear the recording.

ALL: All of the information you provide is voluntary and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Do you have any questions?

NOTE: Written parental consent and verbal youth assent should be obtained before starting the recording. The parent/legal guardian will be escorted back to lobby/waiting area.

If, for any reason, the parent/legal guardian or student does not provide consent or assent to participate, thank the participant for his/her time and end the interview.

NOTE: Students will be administered all items and probes. The duration of each cognitive interview will be 60 minutes.

Part II. Instructions and Generic Probes

The interviewer should ask the participant to read and answer the questions, one at a time. Each question will be a discrete question (e.g. multiple choice). The respondent is not supposed to read the question aloud, but should work on the question in the same fashion he/she would during a NAEP test administration or survey. If a respondent indicates they do not know a word or phrase, the interviewer can provide/read words for the respondent.

Generic probes will include the following questions in the table below. Specific probes may be added following the focus group phase of the study.

Generic Probes for Discrete Items (Multiple Choice and Free Response)

No.	Probe	Instructions for Interviewer
1	Can you tell me, in your own words, what the question is asking?	Ask this probe AFTER the respondent has answered the question.
2	Were there any words or parts of this question that were confusing?  Yes  No	Ask this probe for all discrete questions.
3	What did you find confusing? What could we do to make the question less confusing?	Ask both probes only if the respondent answered YES to the previous question.
4	Would you say it was very easy, easy, difficult, or very difficult to answer this question?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all discrete questions.
5	How could we make it easier to answer this question?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
6	How sure are you that you have answered this question correctly?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all discrete questions.
7	Were you comfortable answering this question?  Yes  No	Ask this probe for all discrete questions.
8	How could we make the question more comfortable to answer?	Ask this probe only if the respondent answered NO to the previous question.

Generic Probes for Matrix Items

No.	Probe	Instructions for Interviewer
1	Look back at the first part of the question, can you tell me in your own words what this question is asking you to do?	Ask this probe AFTER the respondent has answered the entire matrix question. This probe is specifically for the “stem” of the item, not the entire matrix item or a specific option/sub-item.
2	Can you tell me, in your own words, what [option a., b., c., etc.] means to you?	Ask this probe for all options/sub-items. Please note probes 2-6 should be asked together for one option/sub-item before moving on to the next option/sub-item.
3	Did you find any words or part of [option a., b., c., etc.] confusing?  Yes  No	Ask this probe for all options/sub-items.
4	What did you find confusing? What could we do to make [option a., b., c., etc.] less confusing?	Ask both probes only if the respondent answered YES to the previous question.
5	Would you say answering [option a., b., c., etc.] was very easy, easy, difficult, or very difficult?  Very Easy  Easy  Difficult  Very Difficult	Ask this probe for all options/sub-items.
6	How could we make it easier to answer [option a., b., c., etc.]?	Ask this probe only if the respondent answered DIFFICULT or VERY DIFFICULT to the previous question.
7	How sure are you that you have answered [option a., b., c., etc.] correctly?  Very Unsure  Unsure  Sure  Very Sure	Ask this probe for all options/sub-items.
8	Were you comfortable answering [option a., b., c., etc.]?  Yes  No	Ask this probe for all options/sub-items.
9	How could we make [option a., b., c., etc.] more comfortable to answer?	Ask this probe only if the respondent answered NO to the previous question.

Example Student Items—Grades 4, 8, 12 (Cross Grades)

NOTE: The following items and probes are examples of the kinds of questions we may ask students in the cognitive labs. The exact wording of the survey items and probes will be determined based on the findings from the Phase I focus groups.

1. How many parents or guardians do you live with?

2. Next, please say how each parent or guardian is related to you.

a. Adult #1

Mother	Father
Step-mother	Step-father
Foster mother	Foster father
Grandmother	Grandfather
Aunt	Uncle
Other female who is related to you	Other male who is related to you
Other female who is not related to you	Other male who is not related to you

b. Adult #2

Mother	Father
Step-mother	Step-father
Foster mother	Foster father
Grandmother	Grandfather
Aunt	Uncle
Other female who is related to you	Other male who is related to you
Other female who is not related to you	Other male who is not related to you

x. Adult #n

Mother	Father
Step-mother	Step-father
Foster mother	Foster father
Grandmother	Grandfather
Aunt	Uncle
Other female who is related to you	Other male who is related to you
Other female who is not related to you	Other male who is not related to you

Item-Specific Probes

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 8.	General	Required
2	1a. (IF ANSWER IS SOMETHING OTHER THAN 2) How did you decide on your answer?	Specific	Conditional
3	1b. What do you think is meant by “guardian”?	Specific	Required
4	2a. Were there any people not on the list that you think should be there? (If YES) Describe.	Specific	Required
5	2b. What came to mind when you saw, “Other female [or male] who is related to you?”	Specific	Required
6	2c. How about when you saw, “Other female [or male] who is not related to you?”	Specific	Required

3. How far in school did your [FILL RESPONSE FROM 2A] go?

(He/She) did not finish high school.
(He/She) graduated from high school.
(He/She) had some education after high school.
(He/She) graduated from college.
I don’t know.

4. Does your [FILL RESPONSE FROM 2A] work?

Yes.
No.
I don’t know.

Item-Specific Probes

No.	Probe	Type of Probe	Required/Conditional
1	For generic probes for matrix items, see page 4.	General	Required
2	3a. How certain are you of your answer to this question? Explain.	Specific	Conditional
3	3b. What came to mind when you saw “some education after high school”?	Specific	Required
4	4a. What do you think is meant by “work”?	Specific	Required
5	4b. How did you decide which answer to give?
6	4c. If a parent or legal guardian stays at home to care for children, how should someone answer this question?	Specific	Required
7	4d. If a parent or legal guardian volunteers and does not get paid for their work, how should someone answer this question?	Specific	Required
8	4e. If a parent or legal guardian has their own business, how should someone answer this question?	Specific	Required

Part III. General Debriefing and Thank You

Before we finish, I’d like to hear [any/other] thoughts you have about the questions I showed you today.

Is there anything else you would like to tell me about working on the survey questions?

Is there anything that you think could make [this/these] survey question(s) clearer?

[Thank participant for his/her time and provide incentive, as appropriate.]

Distress Protocol

During the focus groups and interviews, all interviewers will be trained to identify and respond appropriately to varying levels of distress. All interviewers will be trained to take notice of the respondent’s emotional state and follow the step by step procedures using the two levels of intervention outlined below. These step by step instructions are designed to respond to the elevated emotional states while also providing the respondent the opportunity to continue with the interview or discussion if she or he so chooses.

LOW TO MODERATE EMOTIONAL STATE: Respondent is showing some signs of emotional distress but there is no perceived or expressed threat to his/her own safety or the safety of others. These signs may include:

Somatic complaints (stomach ache, headache)
Irritability
Physical restlessness (tapping feet, shifting around a lot)
Regressive/self-soothing behavior (thumb sucking, hair sucking, rocking)
Whining
Acting out
Stuttering (sudden onset)

Focus Groups:

If respondent is exhibiting any of these signs of distress, the interviewer will give the respondent time to collect himself/herself by focusing on other students’ responses. If, after a few minutes, the respondent is still exhibiting these signs of distress or if the distress level is increasing, the co-moderator will invite the respondent to leave the room for a few moments and will follow the steps below for cognitive interviews.

Cognitive Interviews:

Step 1: If respondent is exhibiting any of these signs of distress, the interviewer will ask one of the following questions:

How are you doing?
Are you ok?
Do you have any questions?

If respondent indicates she or he is “okay” and does not have any questions, the interviewer will continue with the interview and bring the respondent back into the discussion.

Step 2: If the respondent does not respond or indicates she or he is distressed, the interviewer will ask:

Do you need to take a short break?
Do you need to stop the interview?

If the respondent answers “no” to these questions: The interviewer will continue with the interview/bring the respondent back into the discussion.

Step 3: If the respondent answers “yes” to these questions: the interviewer will allow the respondent to take a short break. Some participants may need a moment to collect themselves and will be willing to proceed.

Step 4: After the break the interviewer will ask: “Do you want to keep going?” If “yes”, the interviewer will continue with the interview and bring the respondent back into the discussion.

Step 5: If “no”, the interviewer will say, “Thank you for participating. I would like to provide you with some names of resources you can call if you would like to talk to someone.” The interviewer will escort the student back to the lobby/waiting area to his/her parent or legal guardian and to receive the incentive.

ELEVATED EMOTIONAL STATE: The interviewer will assess the type of distress and respond using the behavioral indicators listed for each of the three scenarios: extreme distress without intent to harm self or others, expressed self-harm, or expressed harm to others.

Extreme distress without intent to harm self or others:

Uncontrollable crying
Emotional outbursts, including expressions of rage
Respondent isn’t making sense when she/he is talking
Dissociation (can’t remember time/place)
Experiencing flashbacks (like responding to a bad dream)

Expressions of self-harm:

Plans to hurt him/herself

Plans to kill him/herself
Statements indicating she/he might hurt herself
Statements indicating she/he might kill herself
Statements indicating she/he has lost a reason to live or that life has no purpose

Expressions of harm to others:

Plans to hurt someone else
Plans to kill someone else
Expressed statements that she/he might hurt someone else
Expressed statements that she/he might kill someone else
Expressed statements about planning or thinking about using a firearm or other weapon

In all cases the interviewer will stop the interview/remove the student from the focus group discussion. The interviewer will say:

[IN-PERSON]

If any of the above behavioral indicators are observed: STOP THE INTERVIEW/INVITE STUDENT INTO THE HALLWAY and say: “It seems you are upset and it may be helpful to talk to a trained counselor. I am going to provide you with some resources that you can contact on your own. This has contact information for organizations that you can use if you want to talk about any feelings or emotions you experience.”

Counseling Resource List

National Hopeline Network:

Telephone: 1-800-442-HOPE (800-442-4673)

Website: https://www.hopeline.com

Teen Helpline
Telephone: 1-800-TLC-TEEN (800-852-8336)

Website: https://teenlineonline.org/

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Debby E. Almonte
File Modified	0000-00-00
File Created	2021-01-22

National Assessment of Educational Progress (NAEP) Socioeconomic Status (SES) Indicator Items Development Studies

NCES Cognitive, Pilot, and Field Test Studies System

Vol 2 NAEP SES Indicator Items Development Studies 2017.DOCX