Various Demographic Area Pretesting Activities

NCES Cognitive Testing Protocol Interviewer _____

Date _____

Participant Initials _____

Review Informed Assent (2 minutes)

Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF International, a research company that is conducting this study. ICF is conducting a study for the U.S. Census Bureau and the National Center for Education Statistics (NCES). The purpose of the study is to test survey items on bullying for a national survey. Before they use the survey they have asked us to test it to make sure that the questions are clear and understandable. Since the survey is going to be given to young people your age, we have asked you to be one of the people to share your thoughts and opinions about the survey items.

Remember:

• I am an interviewer that is just recording what you are saying, I don’t have an opinion about your answers, so no answer to me is either right or wrong.

• You may stop participating at any time. Even if you decide you do not want to answer any of the questions, you will still receive this incentive.

• The interview will last about one hour.

With this in mind, would you like to continue with the interview?

General Questions About Bullying (8 minutes)

1. What does the word “bullying” mean to you?

[Interviewer: Note, BUT DO NOT PROMPT if interviewee describes or includes the following parts of the bully definition: an imbalance of power; repetition; or unwanted, aggressive behavior]

2. Have you ever seen or heard any instances of bullying? If so, can you describe what happened?

   1. What were the ages of the people involved?

   2. What was the relationship of the two people? Were they friends? How did they know each other?

   3. What was it about this incident that makes you think it was bullying?

   4. Did it happen more than once? If so, how often?

   5. Can you think of any other instances of bullying? (Repeat Q2a through Q2c for up to two other examples.)

3. [If participant indicated in Q2 that they had never seen or heard instances of bullying] Could you describe something that you think would be bullying? (probe for 2-3 examples)

   1. What was it about this incident that would make it bullying?

   2. Can you give me another example of bullying? (Repeat Q3a for up to two other examples.)

Responses to First Survey Item with Vignettes (15 minutes)

Note: Each participant will be shown Question A and either Question B or C. The order of the two questions will be varied between participants. All participants will be shown vignettes 1 through 5; the order in which the vignettes are shown will also be randomized.

During this next part of the interview, I am going to show you a question that you might be asked on a survey. Please don’t answer the question at this point; I just want you to read it over.

Hand participant the first survey item, and give him or her an opportunity to read it over.

4. If you were answering a survey and you came to this question, is there anything that you would find unclear or confusing?

5. Are there any reasons you might have difficulty answering this question?

6. Would you have any questions about this item?

Now I’m going to ask you to answer the question. However, I do not want you to answer based on your own personal real-life experience. Instead, I’m going to ask you to imagine that a specific situation happened to you, and then ask how you would answer the question if that happened to you. O.K.?

Give participant first vignette, and read it aloud to him or her, and then ask the following series of questions:

7. How would you answer this question if this had happened to you? [If question has multiple parts, go through all parts.]

   1. If you answered Yes, why?

   2. If you answered No, why?

   3. (If participant has difficulty answering) You seemed like you weren’t sure if the answer to these questions were “yes” or “no.” Why weren’t you sure?

   4. Obviously, because this situation is imaginary we have left out some details. Is there anything else you would need or want to know about the situation that would have made it easier to answer the question?

Now I’m going to show you a different situation, and ask you to answer the question again. Again, I’d like you to forget about anything that has happened to you in real life, and to think only about this imaginary situation.

Give participant second vignette, read it aloud to them, and ask Q7. Continue with the rest of the vignettes.

Responses to Second Survey Item with Vignettes (15 minutes)

Take back first question, and give the participant the second question. Now I’m going to show you a different question that you might be asked in a survey. Assume that this question is from a different survey, and that you haven’t already been asked the first question you saw.

Hand participant the second survey item, and give him or her an opportunity to read it over.

8. If you were answering a survey and you came to this question, is there anything that you would find unclear or confusing?

Now I’m going to ask you to again answer this question based on the situations that I showed you earlier. Again, please do not answer based on your own personal real-life experience. Instead, when answering imagine that these different situations had happened to you.

Give participant first vignette and then ask Questions 7. Continue with remaining four vignettes.

Follow-Up Questions about Survey Items (10 minutes)

Now I’m going to ask you a few more questions about the two survey items that you have been looking at. Let’s take a look first at the first question I showed you. Give participant the first item.

9. I know I asked you this before, but again—do you think there is anything confusing or unclear about this question?

10. This question describes what “bullying” is in a particular way. Do you agree with this description of “bullying”? Is there anything you disagree with?

    1. Is there anything that you think should be added to this description of bullying to make it more accurate?

11. [For Question A only] Are there any types of bullying that are not reflected in the list of behaviors?

    1. Is there anything in this list that you would not consider bullying?

Now I’d like you to look at the second question I showed you. Give participant second item.

12. I know I asked you this before, but again—do you think there is anything confusing or unclear about this question?

13. This question describes what “bullying” is in a particular way. Do you agree with this description of “bullying”? Is there anything you disagree with?

    1. Is there anything that you think should be added to this description of bullying to make it more accurate?

14. [For Question A only] Are there any types of bullying that are not reflected in the list of behaviors?

    1. Is there anything in this list that you would not consider bullying?

Give participant first item to compare to second.

15. So far we have talked about two questions related to bullying. Show participant the different descriptions. Do these two descriptions seem different to you at all?

    1. If so, how are they different?

    2. Which do you think is a more accurate description of bullying? Why?

    3. If necessary, point out important differences in how bullying is described and ask:

       1. Do you think this difference is important? Which question do you think is easier to understand? Easier to answer? Clearer?

Cyber-Bullying (10 minutes)

Last, I am going to have you look at one other set of survey questions. Show Question D and give a chance for them to review.

16. If you were answering a survey and you came to this question, is there anything that you would find unclear or confusing?

17. Are there any reasons you might have difficulty answering this question?

18. Would you have any questions about this item?

19. Show participant the fifth vignette. Let’s look back at this situation. If this happened to you, how would you answer this question? Why?

20. Would you consider all the things on this list to be “bullying,” or are they not “bullying”?

    1. If so, why? If not, why not? Are any of them bullying?

    2. If only some are bullying, which ones? Why?

21. Is there anything else that should be included in this list?

1. Is there anything in the list that doesn’t belong?

2. Does this list make sense to you? Would you add any others?

Say: Those are all my questions for you. Thank you for helping us today. Is there anything else you would like to add?

Thank you for participating in this interview. At the desk on your way out, you can pick up your stipend. You can also get some materials in case you want to learn more about preventing bullying or have any other questions or concerns about bullying from this interview.

QUESTION A

19a. Now I have some questions about what students do at school that make you feel bad or are hurtful to you. We often refer to this as being bullied. You may include events you told me about already. During this school year, has any student bullied you?

That is, has another student...

(Read each category a-g.)

1. Made fun of you, called you names, or insulted you, in a hurtful way?

2. Spread rumors about you or tried to make others dislike you?

3. Threatened you with harm?

4. Pushed you, shoved you, tripped you, or spit on you?

5. Tried to make you do things you did not want to do, for example, give them money or other things?

6. Excluded you from activities on purpose?

7. Destroyed your property on purpose?

Yes		No
1		2
1		2
1		2
1		2
1		2
1		2
1		2

QUESTION B

In this part, you will be asked about times when you were bullied.

REMEMBER: Bullying happens when someone hurts or scares another person on purpose and the per­son being bullied has a hard time defending himself or herself. Usually, bullying happens over and over.

• Punching, shoving, and other acts that hurt people physically

• Spreading bad rumors about people

• Keeping certain people out of a “group”

• Teasing people in a mean way

• Getting certain people to “gang up” on others

Have you been bullied this school year?

• Yes  No

QUESTION C

The next 2 questions ask about bullying. Bullying is when 1 or more students tease, threaten, spread rumors about, hit, shove, or hurt another student over and over again. It is not bullying when 2 students of about the same strength or power argue or fight or tease each other in a friendly way.

13. Have you ever been bullied on school property?

A. Yes

B. No

14. Have you ever been electronically bullied? (Include being bullied through e-mail, chat rooms, instant messaging, Web sites, or texting.)

A. Yes

B. No

QUESTION D

20a. Now I have some questions about what students that could occur anywhere and that make bad or are hurtful to you. You may include you told me about already.

During this school year, has another student....

(Read each category a-g.)

1. Posted hurtful information about you on Internet, for example, on a social networking like MySpace, Facebook, Formspring, Twitter?

2. Purposely shared your private information, photos, or videos on the Internet or mobile phones in a hurtful way?

3. Threatened or insulted you through email?

4. Threatened or insulted you through instant messaging or chat?

5. Threatened or insulted you through text messaging?

6. Threatened or insulted you through online gaming, for example, while playing XBOX, of Warcraft, or similar activities?

7. Purposefully excluded you from online communications?

Yes		No
1		2
1		2
1		2
1		2
1		2
1		2
1		2

Vignette 1 [bullying, all aspects included]

You and Sonia are in the same friend group that sat together at lunch and rode home on the bus together. You have short, brunette hair and wore glasses, while Sonia has long, blond hair and frequently gets attention from the boys in your grade. When you both sign up to be the lead role in the spring school play, Sonia tells you not to sit with them at lunch anymore because she doesn’t want you to learn any of her secrets to becoming the lead in the play. Sonia also tells you that you can’t sit with her on the bus. You are teary and upset that Sonia has kicked you out of your friend group.

Vignette 2 [no imbalance of power]

You and Dave have been in the same class since 4^th grade. You have never been friends but have worked on a few projects together in class. You are similarly liked in class. One day, you trip over Dave’s foot when you went to turn in a paper. You go sprawling and the whole class laughs. Dave says it was an accident but seems to enjoy the attention the class gives him. The next week, when you are playing basketball together at recess, he elbows you, knocking you down so hard that you lose your breath. You feel embarrassed and frustrated and tell him to knock it off.

Vignette 3 [no aggression]

On Monday afternoon, you use your recess time to hang campaign posters around the school, which read, “Vote for [your name] for Student Council President.” As you walk into school on Tuesday morning you see that most of your posters have been written on. Someone had crossed out the word, in homeroom, “President,” and replaced it with “First Lady.” When you get into homeroom, you find one of your posters taped to the blackboard in the front of the classroom. One of your classmates pointed to a girl in the first row, indicating that Maria, one of the popular girls, had altered the poster. You glare at her as you tear down the poster. “It’s just a joke,” Maria laughs. “Lighten up - I’m gonna vote for you.”

Vignette 4 [no repetition]

Jack is the captain of your track team, and is in the grade above you. One day he snickers when you walk by him during practice. He holds his nose and yells, “You stink!” and runs away laughing. Your other teammates laugh and point at you, but the next day at practice, no one mentions it or treats you differently.

Vignette 5 [cyber bullying, all aspects included]

Tanya is pretty popular. You and her are running for class president. The election is a week away, and Tanya is neck and neck with you in the race. Sara’s friends decide to sabotage you. They create a fake social network page for you, taking a picture from your real profile. For your interests, they write, “partying, making fun of anything Asian, loving myself.” Most of the students in the school are Asian, and rumors start to spread that you are racist. As election day nears, Sara’s friends start to send you texts that say things like “racist” almost every hour.

Page 5 of 14