National Crime Victimization Survey
School Crime Supplement
DRAFT Cognitive Testing Protocol
November 18, 2013
1d: How many different schools have you attended this school year? [if student said “yes” to 1b add: Include your homeschooling as one school]
ISSUES: Only changes since the start of the current school year count as a change in school. So we need to determine if students are confused if they moved from elementary to middle or middle to high school at the start of this year.
PROBES:
Can you tell my why you gave that answer?
What school/schools were you thinking of?
What time period were you thinking of when I asked how many different schools you’ve attended?
What do you think we mean by “different schools”? What kind of change do you think would count as going from one school to a different school?
[if answer was 2+ different schools] Can you talk me through the changes in schools [if needed, ask probes below]
What school were you in at the start of the school year?
When did you change schools?
To what school?
Why did you change?
Did you make any other changes during the school year? [when was that change, etc.]
10. The next questions are about security measures that some schools take. Does your school have:
Security guards or assigned police officers?
Other adults supervising the hallway, such as teachers, administrators, or parent volunteers?
Metal detectors, including wands?
Locked entrance or exit doors during the day?
A requirement that visitors sign in and wear visitor badges or stickers?
Locker checks?
A requirement that students wear badges or picture identification?
One or more security cameras to monitor the school?
A code of student conduct, that is, a set of written rules or guidelines that the school provides you?
ISSUES with 10a/b: Previously response category b only said “other adults supervising the hallway” and did not include examples. This confused some students because they saw “other” as excluding teachers rather than referring to adults “other” than security guards or police officers mentioned in response category a. Examples of teachers, administrators or parent volunteers were added in response category b to clarify.
PROBES:
Who did you think of when I asked about “security guards or assigned police officers”? [if students thought of teachers/administrators/parents AS security guards in response category a because they acted in that role, probe for other wording that would help clarify/distinguish response categories a and b]
Who did you think of when I asked about ‘other adults’ supervising the hallway?
ISSUES with 10e: Previously response category e only mentioned that visitors “sign in.” The addition of wearing badges/stickers reflects the more common security procedure – after signing in, visitors are required to have a visual symbol (badges) in the school building. We want to know if the addition of the wording on wearing a badge/sticker confuses students, since it is now a double-barreled question – asking about both signing in AND wearing badges/stickers. Also, conceptually, we want to know if students know if visitors have to sign in. If visitors don’t wear badges/stickers, would students know they are visitors? Are there any schools where visitors don’t have to wear badges/stickers?
PROBES:
What did you think of when I asked [read response category e]?
Do you know what the school rules are for when adult visitors come to your school?
Do you ever see adults in school who don’t seem to work there?
[if yes] Do those adults wear badges or stickers that indicate they are visitors?
11. If you hear about a threat to school or student safety, do you have a way to report it without having to give your name?
ISSUES: Chandler noted that the original intent of question was whether threats could be reported to someone who could act on the information. Members questioned the word “authority” – do students exclude parents? Is anonymity required, or is the assumption they won’t report without anonymity?
NOTE: It’s unclear what the goal of cognitive testing is. Is it to get at the intent of the original question – whether the student knows the kind of person to report to who could actually do something about the threat? Or is it about anonymity, regardless of whether they think the person they report to can do something about the threat?
PROBES:
[pending clarification]
15. There is a STUDENT at school who…
Really cares about you.
Listens to you when you have something to say.
Believes that you will be a success.
ISSUES: Temkin noted that this item does not have face validity. We need to determine if this question format captures “reciprocal” friendships rather than “unrequited.” Research indicates distribution and quality of friendships is related to protective factors – not having “a friend”. In all research found involving school friendships, only reciprocal friendships were counted.
NOTE: Does the research refer to this particular question (it does not use the terms “friend” or “friendship”), or is this wording designed to address the problems identified with a related question?
PROBES:
Who were you thinking of when I asked that question (you don’t have to tell me their real names – just think of what person or people you had in mind).
Do you think of them as a friend? (why or why not?)
Do you think they think of you as a friend? (why or why not?)
What else is important in a friend to show they like you as much as you like them?
18. Thinking about your school, would you strongly agree, agree, disagree, or strongly disagree with the following...You feel safe in your school
ISSUES: Modzeleski noted school-level surveys are better indicators of presence of security measures. Lerman noted that this is actually about student awareness of security measures; however wording is ambiguous with regard to awareness vs. feeling “safe”. Change in wording and addition of question 16g designed to address this ambiguity. Test correlations between feelings of safety (18) and crime/security measures (10,16,17)
NOTE: It’s not totally clear what the “change in wording and addition of question 16g” is. Is it just the addition of the newly-numbered Q18, or was another prior question also changed?
10. The next questions are about security measures that some schools take. Does your school have:
Security guards or assigned police officers?
Other adults supervising the hallway, such as teachers, administrators, or parent volunteers?
Metal detectors, including wands?
Locked entrance or exit doors during the day?
A requirement that visitors sign in and wear visitor badges or stickers?
Locker checks?
A requirement that students wear badges or picture identification?
One or more security cameras to monitor the school?
A code of student conduct, that is, a set of written rules or guidelines that the school provides you?
16. Thinking about the neighborhood where YOU LIVE, would you strongly agree, agree, disagree, or strongly disagree with the following…There is a lot of crime in the neighborhood where you live.
17. Thinking about the neighborhood where your SCHOOL IS LOCATED, would you strongly agree, agree, disagree, or strongly disagree with the following...There is a lot of crime in the neighborhood where you go to school.
PROBES:
[review answers to Q10; Q16 and Q17].
Refer to Q10: Earlier you mentioned that your school takes these security measures: [x,y,z]. Do you think those measures work to keep you safe?
[if no] What do you worry could happen, even though the school has those security measures?
Refer to Q16: When I asked you about crime in the neighborhood where you live, what were you thinking of? Can you walk me through what specific blocks near your house you had in mind as the “neighborhood where you live”?
Can you tell me more about why you answered the way you did?
Refer to Q17: When I asked you about crime in the neighborhood where your school is located, what were you thinking of? Can you walk me through what specific blocks near your school you had in mind?
Can you tell me more about why you answered the way you did?
Probe to see if student could focus on the difference in Q16 versus Q17 (feeling safe where they live versus go to school)
19. The following question refers to the availability of drugs and alcohol at your school.
FIELD REPRESENTATIVE - For each item ask,
Is it possible to get at your school?
Alcoholic beverages
Marijuana
Prescription drugs illegally obtained without a prescription, such as Oxycontin, Ritalin or Adderall?
Other illegal drugs, such as cocaine, uppers, or heroin.
If “Yes” is marked, ASK - What drugs? (Exclude tobacco products.)
ISSUES: The previous question included a long list of individual drugs; the new item combines the list into two categories of prescription drugs and illegal drugs with a few key examples of each. Issues with the previous extensive list of drugs:
Detail was not reported – all combined as single “drug availability”
Drugs of choice change over time requiring yearly updates
The high percentage of “don’t know” responses to each drug decreases reliability of combined measure
The purpose is to determine if these examples capture the most severe/most common drug activity, or whether additional examples are needed.
PROBES:
Prescription drugs are legal but sometimes people get and sell them illegally, like oxycontin, Ritalin and adderall. Are there other kinds of prescription drugs you know of that are available?
What are some of the other names for those drugs?
Are they very popular?
What about drugs that are illegal, like cocaine, uppers and heroin? Do you know of other illegal drugs that are available?
What are some of the other names for those drugs?
Which drugs do you think are fairly popular now?
What are the most common names for those drugs?
20. During this school year, did you see another student who was under the influence of illegal drugs or alcohol while they were at school?
ISSUES: The focus should be on actual observed behavior – what it is that makes the respondent think the student was under the influence.
PROBES:
Can you tell me more about what you saw?
What made you think the student was under the influence?
What kinds of drugs or alcohol do you think the student was using?
What makes you think that?
NOTE: The sentence “You may include events you told me about already” seems to be referring to events reported in the core NCVS. If so it could be confusing as is since respondents wouldn’t have had the chance to tell us about any events already. Should we (1) drop this sentence for the cognitive testing (2) include an abbreviated version of the core, or (3) explain that there would normally, in a real interview (versus this test) there would have been earlier questions asking about any crime at all?
Two different versions of Question 22 (v1 and v2) need to be tested. For the first round of up to five interviews, ask 22.v1. Use results to refine probes for 22.v2 and use 22.v2 in next round.
22.v1 Now I have some questions about what students do at school that makes you feel bad or is hurtful to you. We often refer to this as being bullied. You may include events you told me about already. During this school year, has any student bullied you?
ISSUES: Determine whether this question captures bullying, or whether the examples in 22.v2 are necessary. Also determine whether the respondent considers the additional two components of bullying (repetition and power differential) as necessary it to be considered “bullying.”
PROBES:
If respondent says yes to Q22.v1:
Can you tell me about what happened?
Did it happen only one time, or more than once? [if one time] Were you afraid it might happen again?
Did the other person have more power than you in some way – for example, were they stronger or bigger than you, more popular, or had more money than you?
All respondents:
Can you tell me what you think of when you hear the word “bullied”?
What kinds of behavior do you think “count” as bullying? Why?
I’d like to read through a short list and for each one can you tell me if you think this would count as bullying?
Made fun of you, called you names, or insulted you, in a hurtful way?
Spread rumors about you or tried to make others dislike you?
Threatened you with harm?
Pushed you, shoved you, tripped you, or spit on you?
Tried to make you do things you did not want to do, for example, give them money or other things?
Excluded you from activities on purpose?
Destroyed your property on purpose?
Do you think of bullying as something that is repeated over and over (or could be), or do you think even if something only happened once and is not likely to happen again that it could be considered bullying?
What about the idea of one student having more power than the other in some way - for example, one being stronger or bigger than the other, more popular, or having more money. Do you think that if two students have about the same power that they could bully each other?
22.v2 Now I have some questions about what students do at school that makes you feel bad or is hurtful to you. We often refer to this as being bullied. You may include events you told me about already. During this school year, has any student bullied you?
That is, has another student...
(Read each category a-g.)
Made fun of you, called you names, or insulted you, in a hurtful way?
Spread rumors about you or tried to make others dislike you?
Threatened you with harm?
Pushed you, shoved you, tripped you, or spit on you?
Tried to make you do things you did not want to do, for example, give them money or other things?
Excluded you from activities on purpose?
Destroyed your property on purpose?
ISSUES: Do the incidents listed in a-g help remind students of additional instances that fall under bullying, or do respondents lose track of the bullying component and report more on non-bullying situations (e.g. friends teasing each other).
PROBES:
If respondent says yes to Q22.v2:
Can you tell me about what happened?
Did it happen only one time, or more than once? [if one time] Were you afraid it might happen again?
Did the other person have more power than you in some way – for example, were they stronger or bigger than you, more popular, or had more money than you?
All respondents:
Can you tell me what you think of when you hear the word “bullied”?
What kinds of behavior do you think “count” as bullying? (Why?)
Do you think of bullying as something that is repeated over and over (or could be), or do you think even if something only happened once and is not likely to happen again that it could be considered bullying?
What about the idea of one student having more power than the other in some way. Do you think that if two students have about the same power that they could bully each other?
23a. When you were bullied this school year, did it happen over and over, or were you afraid it would happen over and over?
ISSUES: This question was designed to align SCS bullying estimates with the CDC definition of bullying. We need to determine to what extent these questions would reduce the percentage of students indicating they are bullied on the SCS. We also need to determine whether the student thinks an event has to be repeated (or fear it could be repeated) for it to “count” as bullying. Also, since this is a new question, determine whether it is generally understood as intended.
PROBES:
Can you tell me, in your own words, what this question is asking?
If something only happened one time, do you think that would “count” as bullying? Why or why not?
Do you think it should count as bullying if it only happened once?
What do you think it means to feel afraid that something would happen over and over?
23b. When you were bullied this school year, were you ever bullied by someone who had more power or strength than you? This could be because the person was bigger than you, was more popular, had more money, or had more power than you in another way.
ISSUES: This question was designed to align SCS bullying estimates with the CDC definition of bullying. We need to determine to what extent these questions would reduce the percentage of students indicating they are bullied on the SCS. We also need to determine whether the student thinks an event has to have a power imbalance for it to “count” as bullying. Also, since this is a new question, determine whether it is generally understood as intended.
PROBES:
Can you tell me, in your own words, what this question is asking?
If something happened between two students but neither one had more power over the other one – they were about the same strength and size, and neither one was more popular or had more money than the other – do you think that would “count” as bullying? (why or why not)?
Would you change your answer to question 22 after hearing Q23a and 23b?
24. During this school year, how often were you bullied?
ISSUES: Based on interviewer feedback, students are sometimes confused as to whether this refers to each way they were bullied, or the total of all bullying episodes (regardless of the way they were bullied).
PROBES:
Can you tell my why you answered the way you did?
Were you thinking of different ways you were bullied, or only the times you were bullied in the same way?
[if the latter] Why did only those incidents come to mind?
Still thinking about all of the times that you were bullied, where did the bullying occur? Did it occur …
In a classroom at school?
In a hallway or stairwell at school?
In a bathroom or locker room at school
In a cafeteria or lunchroom at school?
Somewhere else inside the school building? – Specify
Outside on school grounds?
On the way to or from school such as on a school bus or at a bus stop?
Online or by text?
ISSUES:
25/7: This response category was aligned with other parts of the survey that include to/from school as part of “at school.”
25/8: This is a new response category that reflects changes in school policy; the use of electronics needs to be accommodated into categories of bullying. Responses on category 8 should also be compared to responses on 29a below.
PROBES:
25/7:
Can you tell me why you answered the way you did?
What about being bullied on the way to or from school, including on the bus or at the bus stop. Can you tell me in your own words what that question is asking?
25/8:
Can you tell me, in your own words, what it means to be bullied online or by text?
A little earlier I asked whether you were bullied (re-read appropriate version – Q22.v1 or Q22.v2)
Now I have some questions about what students do at school that makes you feel bad or is hurtful to you. We often refer to this as being bullied. You may include events you told me about already. During this school year, has any student bullied you? [etc.]
Were you thinking of bullying that can happen online or by text when I first asked that question, or just bullying that happens in person?
Did any kind of bullying happen to you online or by text that you did not think about or didn’t mention for some reason back at that earlier question? (Why or why not?)
What about bullying that started at school but then continued online or by text – would you think of the online or text bullying as being related to school in some way, even though it didn’t actually happen at the physical school? (Why or why not?)
And what a schoolmate bullied you online or by text but it never happened at the physical school – do you think you would say yes or no to Q25/8? (Why?)
27. This school year, how much has bullying affected…
(READ STEM AND RESPONSES FOR EACH)
Your school work
Your relationships with friends
How you feel about yourself
Your physical health - for example, caused injuries, gave you headaches or stomach aches.
You in other ways?
ISSUES: The purpose is to determine if this question captures the most serious/common effects of bullying. We may not need the last item on final survey.
PROBES:
Can you tell me why you answered the way you did?
Has bullying affected you in other ways?
In what ways?
How seriously has it affected you?
28. When you were bullied in school this year, did you ever think it was related to...
Your race?
Your religion?
Your ethnic background or national origin - for example, people of Hispanic origin?
Any disability you may have – such as physical, mental, or developmental disabilities?
Your gender?
Your sexual orientation - by this we mean homosexual, bisexual, or heterosexual?
Your physical appearance?
Some other reason?
ISSUES: The purpose of testing is to find ways to evaluate and reduce the frequency of “don’t know” responses. The question only asks if students “think” the incident was related to the categories of race, religion, etc., not whether the student knows for sure that it was related. If “don’t know” is being given because the student feels they don’t know for sure that there was a connection to race, religion etc., explore alternative ways of asking the question (eg: adding wording such as “even if you don’t know for sure, did you ever think it was related to…”). Also, for response category f, we need to determine if this vocabulary (homosexual, heterosexual) is well understood, particularly by younger students. For response category h, we need to determine whether we are missing any significant category, or whether one of the other items needs clarification so students understand what to include there.
PROBE:
Can you tell me why you answered the way you did?
Did you feel you’d have to be certain that the incident was related to one of these categories for you to say “yes”? Or do you think you could say “yes” even if you were not absolutely certain there was a connection?
Response category f: Can you tell me, in your own words, what this means: “Your sexual orientation - by this we mean homosexual, bisexual, or heterosexual?”
What if I had asked it this way: “Your sexual orientation – by this we mean gay, straight, or bisexual?” Are both versions understood equally well?
For item h probe: What other reasons do you think you were bullied?
29a. Now I have some questions about what students do that could occur anywhere and that make you feel bad or are hurtful to you. You may include events you told me about already. During this school year, has another student bullied you online or by text?
ISSUES: Part of change to align with CDC definition: electronic bullying is seen as a means, not a separate type of bullying. With definitional alignment comes a need to differentiate electronic bullying that is under school purview from that which is not. Cognitive testing in 2012 indicated that incidents merge – they start in person, continue online, and vice versa. That testing also indicated that students do not differentiate between types of electronic communication, nor do they restrict a conversation thread to one mode. The essential question is whether cyber-bullying that is part of school bullying is included in answers on 25, or whether we need this separate question. Do students see this question only as a repetition of 25, or are they bringing up other instances? If they did not indicate they were bullied in 25 and never got question 29a, would they have changed their answer about being bullied “at school” if they thought about online/text bullying by classmates as school related? Or, does this confuse students who think if they never use their devices at school the electronic bullying can’t be “at school”.
PROBES:
[if yes] Can you tell me about what happened?
Can you tell me, in your own words, what this question is asking?
Can you give me some examples of what you think of as being bullied online or by text?
[If “yes” at 25/8 and “no” at 29a]: Earlier you had said you were bullied online or by text at school, but you said “no” to this question. Can you tell me why?
[If “yes” to both 25/8 and 29a]: Did this question make you think of bullying incidents different from the ones you already included in the earlier question (25/8)?
What made you think you should [include or exclude] the online/text bullying in that earlier question?
[If “no” to 25/1 and 25/8, and “yes” at 29a]: In that earlier question you did not mention being bullied at school, and you said “yes” to this question. Can we talk about that – what is it about the incidents that made you say “no” to the earlier question but “yes” to this one?
29b. When you were bullied online or by text by another student this school year, did it happen over and over, or were you afraid it would happen over and over?
ISSUES: This question was designed to align SCS bullying estimates with the CDC definition of bullying. We need to determine to what extent these questions would reduce the percentage of students indicating they are bullied on the SCS. We also need to determine whether the student thinks an event has to be repeated (or fear it could be repeated) for it to “count” as bullying. Also, since this is a new question, determine whether it is generally understood as intended.
PROBES:
Can you tell me, in your own words, what this question is asking?
If something only happened one time, do you think that would “count” as bullying? Why or why not?
Do you think it should count as bullying if it only happened once?
29c. When you were bullied online or by text by another student this school year, were you ever bullied by someone who had more power or strength than you? This could be because the person was bigger than you, was more popular, had more money, or had more power than you in another way.
ISSUES: This question was designed to align SCS bullying estimates with the CDC definition of bullying. We need to determine to what extent these questions would reduce the percentage of students indicating they are bullied on the SCS. We also need to determine whether the student thinks an event has to have a power imbalance for it to “count” as bullying. Also, since this is a new question, determine whether it is generally understood as intended.
PROBES:
Can you tell me, in your own words, what this question is asking?
If something happened between two students but neither one had more power over the other one – they were about the same strength and size, and neither one was more popular or had more money than the other – do you think that would “count” as bullying? (why or why not)?
Would you change your answer to question 29a after hearing Q29b and 29c?
30. During this school year, has anyone called you an insulting or bad name at school having to do with your race, religion, ethnic background or national origin, disability, gender, or sexual orientation? We call these hate-related words.
ISSUES: We need to determine if this question is picking up new incidents or if the incident was already reported in Q22 or Q28
28. When you were bullied in school this year, did you ever think it was related to...
Your race?
Your religion?
Your ethnic background or national origin - for example, people of Hispanic origin?
Any disability you may have – such as physical, mental, or developmental disabilities?
Your gender?
Your sexual orientation - by this we mean homosexual, bisexual, or heterosexual?
Your physical appearance?
Some other reason?
PROBES:
Can you tell me why you answered the way you did?
[If 28 was ‘yes’ to any category, and 30 is ‘no’]: When I asked [Q28] you said you thought the bullying was related to [read categories]. And now when I asked [Q30] you said ‘no.’ Why is that?
[If 28 was ‘no’ to all categories, and 30 is ‘yes’]: When I asked [Q28] you said you thought the bullying was not related to [read categories]. And now when I asked [Q30] you said ‘yes.’ Why is that?
For you, what makes being called a hate-related word different than being bullied?
[Section H: Avoidance]
34a. Did you AVOID any activities at your school because you thought someone might attack or harm you?
34b. Did you AVOID any classes because you thought someone might attack or harm you?
34c. Did you stay home from school because you thought someone might attack or harm you in the school building, on school property, on a school bus, or going to or from school?
[SECTION I: FEAR]
Intro: Sometimes, even if you can’t avoid a place, you may still be afraid of what might happen there.
35a. How often are you afraid that someone will attack or harm you in the school building or on school property? (READ CATEGORIES.)
35b. How often are you afraid that someone will attack or harm you on a school bus or on the way to and from school? (READ CATEGORIES)
35c. Besides the times you are in the school building, on school property, on a school bus, or going to or from school, how often are you afraid that someone will attack or harm you? (READ CATEGORIES)
Never
Almost never
Sometimes
Most of the time
ISSUES: Students may not realize the Fear section is asking about a different concept than the Avoidance section, since the questions are quite similar. A new introduction was added to the Fear section to help distinguish the two sections. We need to evaluate whether this introduction was effective.
PROBES:
Now I’d like to ask you about those last three questions. Can you tell me, in your own words, what [35a] was asking?
Do you think of that as being the same or different from [34a]
[same for 35b/34b]
[same for 35c/34c]
| File Type | application/msword |
| File Title | Q1d: When did you change schools |
| Author | Sheldon Rudie |
| Last Modified By | demai001 |
| File Modified | 2013-11-18 |
| File Created | 2013-11-18 |