Download:
pdf |
pdfAppendix F
Cognitive Testing of Questionnaire
Items Summary Report
ED School Climate Surveys (EDSCS):
Violence Questions
Alisú Schoua-Glusberg, PhD
Katherine Kenward, MA
Daniela Glusberg
Findings
Research Support Services, Inc.
December 2, 2015
1
1. Introduction
This report summarizes findings of the cognitive testing of three potential new items to be added
to the ED School Climate Surveys, two for students and one for staff. Research Support Services
Inc. conducted the testing: forty-one cognitive interviews were conducted in Illinois, in the
Chicago Standard Metropolitan Statistical Area between November 7 and 30, 2015.
All respondents worked or studied at a middle or high school1. Ten interviews were conducted
with teachers, either working in high schools or in middle schools. Eleven2 interviews were
conducted with non-instructional staff. Ten interviews were conducted with middle school
students and ten with high school students.
2. Recruiting
Participants were recruited through online postings and flyers targeted at middle and high school
teachers and school staff, as well as parents of middle- and high schoolers. Flyers (shown in
Appendix A) were also handed out to staff in school offices and security guards at the entrance
of schools, as well to students outside of schools. Four schools were visited for flyer distribution:
two suburban middle schools, one suburban high school, and one high school in the north side of
Chicago. An electronic flyer was also posted on Research Support Services' Facebook page, and
staff shared it through their personal networks. Interested candidates had to respond to one of
four telephone screeners, one for students, on for teachers, one for staff and one for principles
(see Appendix B).
2.1. Respondent Characteristics
Of the 41 total participants, 20 were students and 21 were teachers or school staff. Of the 20
students, 15 were enrolled in public schools, 4 in a charter schools and 1 in a private school;
12 were male and 8 were female. The distribution by grade level was: 1 fifth grader, 9 middle
schoolers, and 10 high schoolers. In terms of race/ethnicity, there were 6 Hispanic and 14 nonHispanic. Racial distribution was 4 White, 11 African-American, 1 Asian, 1 bi-racial, and 3 other
(Hispanic). Age distribution was: one 17 year old, one 16 year old, three were 15, five were 14,
four were 13, five were 12, and one was 10 years old.
Of the 21 teachers and other school staff, 6 were male and 15 were female. Among the 10
teachers, 4 taught middle school, and 6 taught high school. The 11 staff included 1 principal, 3
1
Including 5th grade students or teachers in schools that include a middle school.
Right after the planned 20 interviews with staff were completed, one additional principal agreed to be interviewed. Hence a total
of 21 interviews were completed to add another principal's perspective.
2
2
administrative support staff, 2 security staff, 1 detention specialist, 1 school psychologist, 1
recess and after-school monitor, 1 student activities specialist, and 1 tutor specialist.
Teacher Characteristics
ID
Number
Interview
Type
Gender
School Level
School
Type
Teacher
Type
School
Location
Experience
Hispanic?
Race
1001
Teacher
Male
Private
3 to 6 years
Yes
Teacher
Female
Urban
3 to 6 years
No
Other
(Hispanic)
Asian
2012
Teacher
Male
Middle
School
Public
Core
Subjects
Instructional
Staff
Core
Subjects
Urban
1005
Middle
School
High School
Urban
3 to 6 years
No
2014
Teacher
Female
High School
Private
Urban
3 to 6 years
No
3003
Teacher
Female
Middle
School
Public
Non-Core
Subjects
Non-Core
Subjects
Urban
7 years or
more
No
Black or
African
American
3004
Teacher
Male
High School
Public
Urban
White
Teacher
Female
High School
Charter
No
White
5001
Teacher
Male
Charter
No
White
5002
Teacher
Female
Middle and
High School
High School
Yes
White
5003
Teacher
Male
High School
Public
7 years or
more
7 years or
more
7 years or
more
7 years or
more
7 years or
more
No
3006
Core
Subjects
Core
Subjects
Core
Subjects
Core
Subjects
Core
Subjects
Yes
Other
(Hispanic)
Job Title
School
Location
City/Urban
Experience
Hispanic?
Race
3 to 6 years
No
White
City/Urban
3 to 6 years
Yes
Black or
African
American
Native
Hawaiian
or other
Pacific
Islander
Black or
African
American
Black or
African
American
Black or
African
American
Black or
African
American
Black or
African
American
Black or
African
American
Staff Characteristics
ID
Number
2010
Interview
Type
Staff
Gender
2015
Staff
Female
2020
Staff
2021
Public
Public
Urban
Urban
Urban
Urban
School
Level
Middle
School
Middle
School
School
Type
Public
Female
High
School
Public
Student
Activity
Specialist
City/Urban
7 years or
more
No
Staff
Female
High
School
Public
Suburban
7 years or
more
No
3005
Staff
Female
Middle
School
Charter
City/Urban
7 years or
more
No
3009
Staff
Female
High
School
Public
Suburban
7 years or
more
No
3012
Staff
Female
High
School
Public
Administrati
ve Support
Staff
Recess and
after school
monitor
Administrati
ve Support
Staff
Detention
Specialist
Suburban
7 years or
more
Yes
3017
Staff
Male
Middle
School
Public
Principle
Suburban
7 years or
more
No
4002
Staff
Female
High
School
Public
Administrati
ve Support
Staff
Suburban
7 years or
more
No
4005
Staff
Female
High
School
Public
Safety
Officer
Suburban
7 years or
more
No
4006
Staff
Female
High
School
Public
Student
Management
Personnel
Suburban
7 years or
more
No
Female
Public
School
Psychologist
Tutor
Specialist
3
Black or
African
American
White
Black or
African
American
Black or
African
American
Student Characteristics
ID
Number
1002
1004
2004
2006
2013
2019
3001
3002
3008
3010
3011
3013
3015
3016
4001
4003
4010
4011
5004
5005
Interview
Type
Gender
Grade
Range
School
Type
High
School
Student
High
School
Student
Middle
School
Student
High
School
Student
High
School
Student
High
School
Student
Middle
School
Student
High
School
Student
Middle
School
Student
High
School
Student
Middle
School
Student
Middle
School
Student
Middle
School
Student
Middle
School
Student
High
School
Student
Middle
School
Student
Middle
School
Student
Middle
School
Student
High
School
Student
High
School
Student
Female
9 to 12
Charter
Male
9 to 12
Male
Age
School Location
Hispanic?
Race
15
City/Urban
No
Public
15
City/Urban
No
6 to 8
Public
12
City/Urban
Yes
Black or
African
American
Black or
African
American
White
Female
9 to 12
Public
14
City/Urban
No
Female
9 to 12
Public
14
Suburban
No
Male
9 to 12
Public
17
Suburban
No
Female
6 to 8
Public
13
Suburban
Yes
Female
9 to 12
Charter
14
City/Urban
Yes
Male
6 to 8
Public
13
City/Urban
Yes
Male
9 to 12
Public
13
City/Urban
No
Male
6 to 8
Public
12
Suburban
No
Male
6 to 8
Public
14
Suburban
No
Female
6 to 8
Public
13
City/Urban
No
Male
5th Grade
Private
10
City/Urban
No
Male
9 to 12
Public
14
City/Urban
No
Male
6 to 8
Public
12
Suburban
No
Male
6 to 8
Public
12
Suburban
No
Female
6 to 8
Public
12
Suburban
No
Male
9 to 12
Charter
15
City/Urban
Yes
Female
9 to 12
Charter
16
City/Urban
Yes
4
Black or
African
American
White
Black or
African
American
Black or
African
American
Black or
African
American
Other
(Hispanic)
Asian and
African
American
White
Black or
African
American
Black or
African
American
White
Black or
African
American
Asian
Black or
African
American
Black or
African
American
Other
(Hispanic)
Other
(Hispanic)
3. Interviews
Some teachers/staff were interviewed at the schools where they worked, per their
request. Students and remaining staff were interviewed in a few city and suburban locations: a
rented interview room or public space at the Chicago Cultural Center, and a Chicago hotel lobby
where a parent could sit away from earshot but still be able to view their child.
Parents were asked to sign consent forms for the students. Each interview began with an
introduction and administration of a consent form for teachers/staff and an assent form for the
students (shown in Appendix C). Then each respondent was asked to complete a short
questionnaire while thinking aloud and describing how he or she came up with his/her answers
(shown in Appendix D). Once they completed the questions, the interviewer probed the
respondent answers following a semi-structured protocol (see Appendix E).
4. Testing New Item for Teachers and School Staff
4.1. Methods
Testing with teachers and other school staff was focused on a single questionnaire item.
However, to provide context and avoid singling out the specific item of interest, respondents
were provided with a 2-page questionnaire containing a total of 9 questions, among them the one
being tested.
The full set of questions were:
How strongly do you agree or disagree with the following statements about this school?
Mark One Response
28. I can manage almost any student behavior problem.
Strongly Agree
Agree
Disagree
Strongly Disagree
29. I feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
5
30. The following types of problems occur at this school often: physical conflicts among
students.
Strongly Agree
Agree
Disagree
Strongly Disagree
31. The following types of problems occur at this school often: robbery or theft.
Strongly Agree
Agree
Disagree
Strongly Disagree
32. The following types of problems occur at this school often: vandalism.
Strongly Agree
Agree
Disagree
Strongly Disagree
33. The following types of problems occur at this school often: student possession of
weapons.
Strongly Agree
Agree
Disagree
Strongly Disagree
34. The following types of problems occur at this school often: sexual assault or violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
35. The following types of problems occur at this school often: physical abuse of teachers.
Strongly Agree
Agree
Disagree
Strongly Disagree
6
36. The following types of problems occur at this school often: student verbal abuse of
teachers.
Strongly Agree
Agree
Disagree
Strongly Disagree
Question 34 was the focus of this testing. However, probing also covered questions 28, 30, 3, 35
and 36, to provide context and get the respondents used to answering probes. Interviews were
short, rarely longer than 30 minutes.
Findings
Question 34 read as follows:
34. The following types of problems occur at this school often: sexual assault or violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
Two respondents marked Agree, ten respondents marked Disagree, and nine respondents marked
Disagree Strongly.
4.2. Testing Goals
Testing of this item and the corresponding cognitive probing aimed at answering the following
questions:
Did respondents understand the question as intended?
Were they thinking of sexual assault and sexual violence or were they thinking of
violence in general and sexual assault separately?
Were they thinking of their school as they answered?
Why did they choose one response category over another?
7
4.3. Overall Question Interpretation
Responses to the question varied depending on interpretation, as did response strategies. Twelve
respondents interpreted the question as referring to sexual violence, including sexual assault,
while seven thought the question was asking about two distinct types of things: violence in
general (non sexual in nature) and sexual assault. One respondent asked initially if the question
referred to sexual violence or violence in general; two others asked a similar question during
probing. Another recommended adding 'bullying' to the question (which fits with his
interpretation of violence in general).
In probing, interviewers asked what the respondent thought was meant by 'sexual assault.'
"Rape," "molestation," "inappropriate touching" were the most common inclusions in the
definitions elicited. A few respondents indicated they were thinking of student on student acts,
but also staff on student, or student on staff incidents.
Of the two respondents who marked Agree, one was a teacher who understood the scope of the
question but said she would broaden it to include "inappropriate" music: "Sometimes they
(students) don't consider it inappropriate; if I feel uncomfortable with the activity, I consider it
assault. I think sometimes the girl doesn't want to say anything about it." Upon further probing
she explained: "We have very strict policies about what they (students) can say so they will call
out parts of songs." She felt this is something the boys do that is demeaning to the female
students, but they do not want to report the males to avoid getting them in trouble.
The other respondent who answered 'Agree' did so treating the 'or' as separating two different
kinds of things: on one hand violence (non-sexual) and on the other sexual assault. While she
said sexual assault does not occur, her answer was affirmative because she was answering about
violence in general.
“I agree. Sexual assault, no...but the violence, like I said the students they can be playing
and then they can start fighting each other, or hit each other. You know, they end up
getting suspended because they have an issue with keeping their hands to themselves.
And more so, I would say 6th to 8th graders than it is the younger kids, because the
bigger kids for some reason they are so angry, so if somebody says something to them
they get mad and they want to fight instead of talking.”
The respondent clarified that if the question had asked only about sexual assault she would have
answered “strongly disagree” because: “We haven’t had any issues with sexual assault at the
school.”
For another respondent (case 2015), the question was only asking about physical aspects of
violence or assault. Since in the school "there aren't physical altercations; most of it is verbal,"
the respondent marked Disagree. However, the same respondent, later said "Just as violence can
be emotional or physical, but they are both violence."
8
A respondent who marked Disagree, explained his response as taking into account both violence
in general and sexual assault, and providing an answer that would cover both types of incidents.
“If it was just sexual assault I would have answer ‘strongly disagree,’ but putting
violence out there made me just put disagree because the violence does occur sometimes.
And sometimes and often are definitely two different things, but looking at the question
with ‘often’ again, makes you ‘disagree’.”
Upon further probing, the respondent said that if the question had asked only about sexual assault
she would have answered Strongly Disagree because:
“Inappropriately touching maybe a male to a female or vice versa, it can go both ways,
but I think that’s where it starts.” – “That just doesn’t happen in my school, as far as I
know.”
Some respondents wondered if the question referred only to incidents inside the school or also on
school grounds, or in the neighborhood around the school. One respondent noted that:
“Another thing I’m thinking about too is for these questions maybe clarifying like do
they mean literally like in the school or is it going to count like outside of school, like
after school when kids are hanging out or…a lot of times we hear about things happening
out in the school like in the neighborhood parks and stuff, and then it's still a
repercussion for the kids at school but it’s not necessarily happening in the school.”
[PROBE: Does the question make you think they mean only IN the school?] “Yeah, it
makes me think it's in the school, but I guess my mind is leaning towards things I hear
happening right outside of school or right after school I guess.”
And in the words of another respondent:
"No one has been sexually assaulted here in the building, I just can’t foresee any—I
mean it could possibly happen but I can’t foresee it happening not on day, evening or
night shift, I just can’t foresee it happening—happening in the school, outside the school
I can’t speak. Inside the school I have to disagree totally that is not happening here."
However, generally, respondents felt that frequency for something to be considered to occur
'often' depends on the type of event. For an act that is more serious or more uncommon, such as
rape, 'often' may be just "more than three times a year." Whereas, for a common or less serious
act like verbal altercation, 'often' may be interpreted as "every day."
One respondent indicated that, for a question like Q36 R, he would consider "often" once a
month or 12 times a year for this type of occurrence, whereas for Q34, he considered "often" 3 to
5 times a year.
Thinking of sexual assault, most respondents said they would consider anything over 3 to 5 times
a year to be 'often'.
9
One single respondent gave an absolute response: she defined 'often' as something "happening a
lot of the times that the school is in place... In a week, probably 3 (days) out of the 5.”
4.4. Interpretation of 'Sexual Assault and Violence'
As indicated above, one respondent broadened the definition to include things like "inappropriate
music." However, for most teachers and staff, interpretations tended to focus on inappropriate
touching.
“If you touch somebody inappropriately, touching someone if they don’t want to be
touched, like in their private areas or anywhere where you are not supposed to touch
them. That’s what I think about when I think of sexual assault.”
"Sexual assault means physically, inappropriately touching someone against their will. I
think that for those kids it’s inappropriate because they are in the school property and
they are under age.”
"It could be heterosexual or transgender or homosexual. Could be threats or stalking,
inappropriate contact between faculty and student. Violence and could be verbal as
well as physical."
"Because it's all together like that, to me that's still referring to a violent act that has to
do with sexuality. So for example it could be…related to someone being physically hurt
or harmed because of their sexual orientation."
One of the respondents who separated violence from sexual assault explained the distinction:
“Two different things. Sexual is sexual, rape. Violence, violence, beat, knock somebody
out…Then you could put them as both because it could happen the same way you could
be sexually assaulted and beaten at the same time."
Without exception, all respondents were thinking about the school in which they currently work.
Those working in more than one school were asked to focus their answers on the school where
they work most hours. They did so without any noticeable difficulty.
4.5. How Often Is 'Often'
What constitutes 'often' for respondents varied to some degree, but generally they believed that
the type or seriousness of an incident determines how often is 'often.'
10
One respondent who treated the question as double barreled (asking about violence in general
and also about sexual assault), illustrates the relative meaning of 'often.' For her, "it depends": for
sexual assault, more than three incidents in a year would be 'often' whereas for a verbal
altercation between a staff member and a student, twice a day might be 'often.'
Another respondent thought of 'often' as "something that is continuously going all the time and
we don’t have that problem here…it would be like on an occasional basis…it’s very rare that that
would happen here." For this respondent, 'often' referring to sexual assault might be three to four
times ever. "Often for sexual assault would be different than finding drugs—four or five times in
a month would be often for finding drugs in school for example, one instance would be often for
sexual assault…often to verbal assault on staff would be every day."
4.6. Precision of Response Choices
Interviewers probed on why the respondents chose a specific response category rather than an
adjacent one. This is a helpful approach to understand how respondents interpret the categories.
From their narratives, it emerged that respondents tended to answer Disagree when they had not
heard of any specific incidents, which gave them the impression that such things do not happen
often (or else they would have heard). However, in some cases, they felt unsure that it does not
happen at all simply because they are unaware, and this stopped them from answering "strongly
disagree."
Disagree answers came generally from those who have not heard much, or have not heard of
incidents inside the school, or only of an odd incident, but nothing they would see as happening
often. Among those who selected Strongly Disagree, while not systematically different, a few
articulated stronger reasons: "no incidents in 20 years," "none inside the school," "never heard of
a rape."
4.7. Recommendations
If this item is included in the survey, we recommend that it be rephrased to avoid the different
interpretations regarding 'sexual assault and violence.' All other concepts in the item worked
well in the testing. The rephrased item would read:
The following types of problems occur at this school often: sexual assault or sexual
violence.
11
5. Testing Items for Middle and High School Students
5.1. Methods
Testing with students focused on two items. However, to provide context and avoid singling out
the specific item of interest, respondents were provided with a 2-page questionnaire containing a
total of 10 questions, among them the two being tested. The questions were:
13. It is easy to talk with teachers at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
14. My teachers care about me.
Strongly Agree
Agree
Disagree
Strongly Disagree
15. At this school, there is a teacher or some other adult who students can go to if they need
help because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
16. My teachers make me feel good about myself.
Strongly Agree
Agree
Disagree
Strongly Disagree
30. I feel like I am part of this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
12
31. I feel socially accepted.
Strongly Agree
Agree
Disagree
Strongly Disagree
32. I feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
33. I feel safe going to and from this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
34. Students at this school feel unsafe because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
35. I sometimes stay home because I don’t feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
Questions 15 and 34 were the focus of this testing. Respondents were asked to self-administer
the questions. In addition to the two questions tested, probing was done on questions 13, 14, 32,
33 and 35, to set the stage, provide context, and train the students on probe answering. The
interviews were short, as those of the adults, most often under 30 minutes.
13
5.2. Question 15 Findings
Question 15 read as follows:
15. At this school, there is a teacher or some other adult who students can go to if they need
help because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
All of the high school students marked either Agree (n=4) or Strongly Agree (n=6). Among the
students in middle school, 3 strongly agreed and 4 agreed. In addition there were three cases
where the student marked Disagree or Strongly Disagree.
5.2.1. Testing Goals
Testing of this item and the corresponding cognitive probing aimed at answering the following
questions:
Were the students thinking of their current school?
Did “some other adult” include adults outside of school?
Is “go to” literal or figurative to them?
Were they including violence that happened outside of school?
What was the student’s level of comfort with this question?
Why did they choose one response category over another?
5.2.2. Overall Question Interpretation
The two key concepts in this question are 'sexual assault' and 'dating violence.' Most high school
students interpreted 'sexual assault' as referring to rape plus other kinds of physical assault of a
sexual nature. One thought only of rape and 2 only of inappropriate and unwanted touching. The
definitions elicited included:
"Unwanted sexual contact."
"Like by being touched or something, the wrong way or, you know, by someone that
shouldn’t be touching you.”
14
"I think it means like if you were violated in some sort of situation or someone
touched them."
"Physically touching someone without their permission."
“Like being harassed or raped or something. Being forced to do something you don’t
want to do basically.
A smaller proportion of the middle school students mentioned rape explicitly, only 3 of the 10.
The remainder spoke of unwanted touching, grabbing, or kissing.
The phrase 'dating violence' was interpreted more dissimilarly in the two age groups. All 10 high
school students referred to violence within the context of a dating relationship, where one or both
persons are physically violent to the other. A couple of students also mentioned screaming or
being verbally violent. None of the students interpreted dating violence as including forcible
sexual behavior.
Among the middle school students, there were a few who did not understand the concept of
dating violence as intended. Two students guessed it might mean "forced dating." References to
physical fights or to the girl in a couple getting hit or punched were common, but also, without explicit
mention of sexual violence, several referred to unspecified "abuse."
Both middle school and high school student respondents were thinking of their current school as they
answered the question, with no exceptions.
5.2.3. The Concept of 'Some Other Adult'
The majority of the high school students only referred in their response to adults in their school.
Upon probing several mentioned that the question asks specifically for someone 'at the school.'
However for 2 respondents, 'some other adult' included adults outside of school.
"It can be any adult that you are close to."
"It could include other people outside of school...(Someone who) could help you by
talking to your parents or helping you find a solution."
Students said they were thinking mostly of teachers, counselors, and social workers. There were
some single mentions of "janitor," "security," "coach," "any school staff," "other parents in the
school," and "principal."
Among the 10 middle school students, 6 were only thinking of adults in the school, while 2 also
included family members. As for school personnel, they also volunteered "peer mediation
teacher," and "social worker." One student was unsure if the question aimed to include or
exclude adults outside the school.
15
5.2.4. Concept of 'Going to'
One goal of testing this item was to know if the students might limit their answer by interpreting
'go to' literally, that is, thinking only of adults they can go in person to meet and talk, and
excluding adults they can contact in other ways, such as by phone or by email.
Both high school and middle school students interpreted 'go to' as intended, not limiting it by
mode of communication or ability to meet in person.
5.2.5. Including Incidents Occurring outside of School
The question is not specific regarding where the sexual assault or dating violence in question
would occur. As intended, all but one of the high school students included in their thinking any
incident that could happen either in school or out of school.
"No, I think it's asking if it's happening period, and you don't feel comfortable telling
your parent or somebody in your family, is there someone at school that you feel
comfortable telling."
“It could be happening somewhere else. It would probably most likely be happening
somewhere else because students at my school mostly feel safe so, I don’t think things like
this would take place at school. It could happen anywhere.”
The one student who thought only of incidents occurring out of school did so simply because
she could not imagine such things could happen inside the school:
“Something that’s happening outside of school most likely….because it’s not really much
you can do in my school, it’s really strict and like they don’t go for stuff like that."
For one of the middle school students, the question was not "exactly clear" as to where the sexual
assault or dating violence would occur, but for the larger number (n=5) it could refer to either
inside or outside of school. Two more thought the question was only about in-school incidents,
and one thought it was only about things happening outside of school (“Most likely
outside…because that’s the time where that mostly happens.”)
5.2.6. Students Level of Comfort with This Item
Interviewers observed the students while they completed the questionnaire and -- in some cases - expressed their thoughts. None of the respondents was observed to be uncomfortable upon
reaching this question, while they self-administered the questionnaire.
16
However, there was some degree of discomfort in a few cases when having to answer the
cognitive probes, in particular, when asked for a definition of sexual assault. By observation of
the interviewers and responses to other probes, there were 2 respondents that appeared less than
fully comfortable discussing this question. The interviewers marked them as 'somewhat
comfortable.'
"Felt like it would always be somewhat difficult to go to an adult about these issues."
Middle school students appeared generally very comfortable with the question. Six of the ten
respondents answered the probes comfortably.
5.2.7. Choice of Response Category
As mentioned above, all of the high school students marked either Agree (n=4) or Strongly
Agree (n=6). Among the students in middle school, 3 strongly agreed and 4 agreed. In addition
there were three cases where the student marked Disagree or Strongly Disagree.
The rationale for selecting agree or strongly agree was well articulated and clear among the high
school students. In schools where affirmative steps have been taken to let students know who
they can go to with these types of issues, this was an easy question to answer.
"I said yeah because at the beginning of the school year they had a seminar or whatever
for if a problem like that were ever to come up to…they gave you instructions on what to
do. They tell you to go to a teacher that you know…and it's your counselor that helps you
with that."
"They always tell us if there is any problem we can come to them."
"Because there are like 10 counselors, and they are all willing to help anyone in any
situation. And also the teachers. We develop strong bonds with the teachers because they
really care about you."
"Because they have people specifically assigned to help with situations like this."
“Because we have counselors, it is set up if your last name is like A to G you go see this
counselor and so and so… With the counselors you can go to them and talk to them with
confidence, and they are there to help you through situation like sexual assault and
dating violence, and if it is really a problem they reach out for you so you don’t have to
because sometimes you are scared and you don’t know what to do. That is why the
counselors are there to help you through situations like these that are affecting you in
school or outside.”
17
“I chose that answer because I said before that it’s a big family. So, there aren’t a lot of
students there, it's just freshmen this year, so it’s really easy to go up to a teacher and ask
for advice and all the teachers are telling you, oh, if you need help with something come
to me or if it’s something private just tell me so I can help you. Because if you have a
problem that’s personal and then it's affecting your work, they can tell.”
“I said strongly agree because we have four counselors I believe, at the school and
they’re all really nice. You only could get pulled out of class if like you really need to
talk to them, and your talk is private no one else hears about it and your information is
not spread through the school…you can talk to them about anything and they’ll give you
advice or help you.”
"Because there are so many people, and they are always willing to listen."
Even when the students do not feel the school has been proactive, there are trusted adults they
feel they can go to:
"There is a teacher at my school that you can always go to talk to about anything."
“Not like everyone cares, not everyone you can trust like they would do something. Like
you can tell a teacher and they can do something but, later on they don’t really do it. Like
if you tell someone it has be someone you really trust. Basically, not all people care but
some do.”
One of the students explained why he chose Agree rather than Strongly Agree:
"I agreed because even though I am not necessarily a result of sexual assault or dating
violence I am sure some people in the school have gone through that...I may know a
couple of people or a couple of stories, but I am not directly related to anybody I know of
that has experience sexual assault or dating violence specifically, so I am not quite sure
the aspects of the school that support that, so I just said agree because I know it's there,
but I am not sure it's definitely there."
None of the middle school students knew of any such incidents ever happening to students in
their schools. They answered from their knowledge of school resources. The narratives elicited
were mostly consistent with the response each gave to the survey item.
"I agree because there are social workers at my school that you can go talk to about
violence or assaults, bullying, school work or problems at home."
"We have peer mediation and we have deans and stuff, so like if anything ever happens
we have cameras in the hallways and they can hear things in classes so if they see
something happening they can catch it on tape. People could tell the dean and the dean
could take over the situation and like suspend or expel or what they need to do."
18
"We had a lot of problems last year, and we had the social worker that came she helped
everyone get through stuff. She let them come to her class at different times and periods."
"I know there is this one teacher that everybody goes to that helps with their problems."
Two respondents selected Strongly Disagree or Disagree because of the lack of a specific person
in their school designated to deal with such issues.
"I disagree, not many people go to counselors because of that and there isn't really a
counselor that does stuff like that."
“Because we don’t have one and that never happened at our school.” (By "don't have
one," the respondent meant a counselor specifically for sexual assault issues.)
Finally, one respondent answered Question 15 inconsistently, given the answers he provided to
probes. At question 14 he said that the teachers at his school really care about the students and
cited that one of his teachers specifically mentioned in class that students could come to talk/see
him at any time. At Question 15, he answered Disagree. He paraphrased Question 15 as "Is there
an adult you can turn to if you need help for any reason?" The interviewer noted that this
respondent seemed to not have fully understood the question initially. First he said the question
was about seeking "advice on dating" but then expanded his definition to include "needing help
not just about dating but for any reason." The respondent seemed confused by the question.
5.2.8. Recommendations
The concept of dating violence may be difficult for some middle school students, as indicated in
the findings. Otherwise the item appears to function well. A different wording or an explanation
of what dating violence means might be useful to assure all are answering about the same type of
incidents or situations.
5.3. Question 34 Findings
Question 34 read as follows:
34. Students at this school feel unsafe because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
19
The distribution of responses followed an almost identical pattern among high schoolers as
among middle schoolers. Middle school students selected Disagree in 3 cases and Strongly
Disagree in 7. Among the high school students, 6 selected Strongly Disagree, 3 selected
Disagree, and 1 answered Agree.
5.3.1. Testing Goals
Testing of this item and the corresponding cognitive probing aimed at answering the following
questions:
How did respondents interpret this item?
How did they decide on their response?
What does “unsafe” mean to them?
Why did they choose one response option over another?
Were they thinking of both sexual assault and dating violence?
5.3.2. Overall Question Interpretation
The probes on interpretation of this item elicited narratives that showed consistency in how
respondents answered the item. The one high school student who answered Agree, did so out of
general societal awareness, rather than from anything that is happening at the school:
"Just because I don't see it directly relating towards me, doesn’t mean it's not there,
because I know that sexual assault and dating violence is a big problem that has been
emerging with teenagers so I know that not all the relationships in the school are 100%
safe and don't have any of these things or components in them. That's why I put agree."
Other study participants took an opposite approach. The fact that they have no awareness of such
cases was generally taken as evidence that it cannot be such a big problem in their school. Their
reason for disagreeing with the statement about students feeling unsafe was based on the
rejection of such issues ever happening for students at their school.
“To my knowledge there haven’t been any acts of sexual assault or dating violence at
[Name of school], so as I see it, I strongly disagree that it is a problem. The principal
normally has assemblies or meetings with the entire school if something is a problem,
and he announces it. We haven’t had anything like that, so I doubt that that’s a safety
problem for the students at our school.”
“I strongly disagree with this because…I feel like it’s a big family at the school…I feel
really comfortable talking with my teachers, I don’t have problems with any students so
it’s just like, I just strongly disagree with it.”
"I have not heard anything about any sexual assault or dating violence. Like I don't see
bruises on people's faces, so I am pretty sure it's safe in terms of that. There may be
fights, but not in terms of those two topics"
20
"I have never heard of anyone being sexually assaulted or like dating…being involved in
dating violence, but also there could be people that I have never heard of that could have
experienced this, so I disagree. But could agree, if there were more information."
Others more clearly were addressing the issue of feeling unsafe:
"Pretty much if you know someone you know they are not going to hurt you or put you in
danger."
“Because it is like some teachers there would help or any other guardians or parents
there would help. They would probably give you advice or so or ask who is doing that or
how would they help, people would call the police or so to help. They would do
something to make you feel better.”
“I chose that answer because the school, no one really...the students know not to bully
anyone or anything because you will get in serious trouble for bullying or sexual assault
or anything, cause most of the students there they’re not, they’ll go talk to a counselor so
they’ll let them know right away and the person will get in trouble most likely.”
Middle school students answered with a similar range of logic responses:
"I disagree because if you are in that situation, teachers probably help, but it kind of
depends because if you don't want to tell a teacher and you want to keep it to yourself
and a teacher is just doing their job and they don't see any signs or anything then it
probably wouldn't be safe."
"I disagreed because not a lot of stuff happens at my school. It's like a boring school. So
not a lot happens, we only had a couple of fights and a couple of sexual assaults that
really weren’t."
"Nothing wrong ever goes down at my school. Some of the boys 'play fight' but no one
gets hurt."
"We don't have any problems in our school with sexual assault." "They are asking about
whether people have problems with teachers or students touching them but we don't have
that."
“Because that never happened around my neighborhood, because my neighborhood’s a
really good neighborhood.” (What do you mean ‘that never happened’ What?) “You
never heard of anything on the news that causes any sexual assault or dating violence.”
"Because everybody at my school is pretty chill and don't touch each other. We all just
talk to each other, there is no dating violence and like people rarely date at my school
and no one really touches each other, no sexual assault, we can't even hug, so there is no
sexual assault either."
21
"Not many people date at my school."
"This usually don't happen at our school, so people should not feel unsafe about it."
It is worth mentioning that some respondents answering Disagree indicated that they would have
answered Strongly Disagree; however, not knowing how other students truly feel, they were not
totally sure none felt unsafe. They felt they could not confidently speak for others.
5.3.3. Meaning of Unsafe
Question 34 asks if students at this school feel unsafe because of sexual assault or dating
violence. This could be interpreted as students who attend that school feeling unsafe or as
students feeling unsafe in the school. Generally respondents interpreted in this latter fashion,
which may not necessarily be the item intent.
The high school students' narrative responses appeared to remain focused on the topic of sexual
assault or dating violence:
"Unsafe as in there is a person at the school that could harm you or do these things to
you."
"Feeling unsafe as in you feel like you don't have anybody to turn to maybe…basically
you don’t feel like the school has provided resources to keep you from an issue in the
school at that time."
"I think they mean that the person might feel, like they might be hurt again by the same
person or other people. They might be blamed or something like that."
"You probably don’t feel like you can really trust, like you don’t feel safe, you don’t feel
protected or so. Or you don’t trust anyone or you feel uncomfortable.”
“Maybe helpless, unprotected because when you come in there and you haven’t talked
about these problems, and if there is a problem and you are coming and you know that
the person who you are having an issue with is there, and no one else knows, then they
can make you feel like you are helpless or defenseless or unprotected because you have
not reached out and now this is a problem for you… If it was outside, now it is a problem
for you at school too.”
"As in they would be scared to go to school or like they were afraid of something at the
school they would feel like they would not want to jeopardize themselves by going in the
school."
22
“They…feel like there’s no one there for them…they have to be careful about their
surroundings.”
"I guess you are scared in a sense to show people. Like maybe they wear long sleeves
because maybe they have a bruise on their arm, so they are constantly pulling it down."
The middle school students provided similar narratives but also brought up bullying and assault
other than sexual or dating related:
"Do you feel unsafe, like do you feel comfortable telling anyone. Or are you nervous to
tell anyone because you think they won't help or they will probably embarrass you
because of your situation. It is kind of asking do you feel comfortable talking to anyone."
"If they feel like something bad is going to happen to them or like someone could get
jumped or something."
"When you feel like someone is going to hurt you after school or in the bathroom."
“They mean as in if they feel as if they will be assaulted, if they would experience those
bad things, which some people do.”
“Is there any bullying or people getting hurt in the school.”
5.3.4. Considering Sexual Assault, Dating violence, or Both
High schoolers were all thinking of both sexual assault and dating violence. Seven of the middle
schoolers also thought of both, while it was not clear what one student included or not, and the
other was only thinking of sexual assault.
5.3.5. Recommendations
Before deciding on adoption of this item, consider whether the intent matches the interpretations
elicited in terms of where the students feel unsafe, and from whom do they feel unsafe.
23
Appendix A:
Flyers
24
25
26
27
Appendix B:
Screeners
28
Recruitment Script for Student/Parent Cognitive Interviews – OMB# 1850-0803
School Climate
Student/Parent Screener
Hello. My name is [XXXXXX] and I work for Research Support Services. We are
conducting a brief but important study about students’ experiences at school. Your
participation is voluntary, and you will receive no sales pitch or follow-up calls based on
your participation. I’d like to tell you a little bit about the study first; is now a good time?
(YES OR NO AND CALL BACK LATER)
Screener
ID_____________
Date_____________
Recruiter__________
____
We are looking for students to help us understand how they interpret survey questions asking about the
student’s school. These survey questions are part of a new suite of school climate surveys that are being
developed by the U.S. Department of Education. The objective of the current study is to assess the clarity
of the survey questions, and we need help from students to do that. The information gained from the study
will be used for potential revisions to the survey questions. Research Support Services (RSS) will be
conducting the interviews. I’d like to ask you a few questions to see if you qualify for the study.
i. What is your name? ____________________
1. Do you have a child in your household who is currently enrolled in middle school or high school?
a. Yes
b. No (THANK AND TERMINATE)
2. What is the name of the school this child is currently attending?
________________________________________
IF PARENT HAS MORE THAN ONE CHILD IN MIDDLE/HIGH SCHOOL ASK THEM TO REFER TO THE ONE WHO
HAD THEIR BIRTHDAY MOST RECENTLY.
3. Is this a public school, private school, charter school, or some other type of school?
a.
b.
c.
d.
e.
Public school
Charter school
[ATTEMPT TO RECRUIT A MIX]
Private school
Other type of school (SPECIFY: ___________)
Don’t know
29
Recruitment Script for Student/Parent Cognitive Interviews – OMB# 1850-0803
4. What grade is this child currently in?
a. Grade 5
b. Grades 6-8 (ATTEMPT TO RECRUIT ~2 5TH GRADE; THEN A MIX OF THE OTHER 2
CATEGORIES)
c. Grades 9-12
Other (THANK AND TERMINATE.)
5. How old is this child?
RECORD AGE
5a. And is this child male or female?
a. Male
b. Female
The next few questions are for classification purposes only.
6. Is this child Hispanic or Latino?
a. Yes
[RECRUIT A MIX OF YES/NO]
b. No
c. Prefer not to answer
7. Which of the following categories best describes this child’s ethnic or racial background? (You
may select one or more races.)
[RECRUIT A MIX OF RACE/ETHNICITY BASED ON RESPONSES TO Q6/7]
a. American Indian or Alaska Native
b. Asian
c. Black or African American
d. Native Hawaiian or other Pacific Islander
e. White
f. Prefer not to answer
8. What kind of area do you live in? (READ LIST)
a.
b.
c.
d.
City/Urban
Suburban
Rural
Don’t know (SPECIFY LOCATION)______________________________________
We are conducting a study with students about new survey questions that ask about their experiences at
school. If your child qualifies, do you give your permission for us to invite him/her to also participate in a
30
Recruitment Script for Student/Parent Cognitive Interviews – OMB# 1850-0803
60-minute interview? Your child will receive $25 to thank them for their time and participation, and if
you bring your child to and from the interview, you will also receive $25.
o
o
Yes
No [THANK AND TERMINATE]
CHILD SCREENING
At this point, I’d like to be able to ask your child a few questions, during which you are welcome to stay
on the line. Is he/she available to speak with me at this time? (IF NOT, GET A RE-CALL TIME AND CALL
BACK)
What is your child’s name?
RECORD NAME_________________________________
Again, you are welcome to stay on the line, but please allow your child to speak for him/herself. Thank
you.
SAY TO STUDENT:
Hi (student’s first name). My name is ___________and I have been speaking with your (mom/dad/other)
for a few minutes.
As I explained to your (mom/dad/other), I work for a research company and we are conducting an
important study about how things are at school. These questions are part of a survey of what students like
you think about their school. We are looking for students who can help us understand what they think
about these survey questions, so we can help to make the survey better. Right now, I’d like to talk to you
to find out if you are able to participate in the study. There are no “right” or “wrong” answers to any of
these questions, and whatever you say is OK. OK?
Great.
31
Recruitment Script for Student/Parent Cognitive Interviews – OMB# 1850-0803
ASK THE FOLLOWING QUESTIONS OF THE STUDENT:
1. What grade are you currently in?
________
2. What type of school do you currently attend? (READ LIST)
a.
b.
c.
d.
Public school
Charter school
Private school
Don’t know
3. What is the name of your school? (RECORD)
_________________________________________________________________________
4. Please tell me what your favorite subject is in school and why. (WRITE VERBATIM)
NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT STUDENTS IN THIS STUDY BE ARTICULATE. IF
STUDENT CANNOT OR WILL NOT EASILY GIVE A ONE-TO-TWO SENTENCE UNPROMPTED ANSWER IN WELL
UNDERSTOOD ENGLISH, PLEASE TERMINATE.
5. Do you think you’d be comfortable coming to an interview to talk with an adult you haven’t met
before about your opinions of some questions about your experiences at school? [ IF CHILD IS A
MINOR, ADD: Your parent will be coming with you. IF NOT A MINOR: A parent or friend can come
to the interview, if you wish.]
SELECT ONE
a. YES
b. NO [THANK AND TERMINATE]
32
Recruitment Script for Student/Parent Cognitive Interviews – OMB# 1850-0803
INVITATION
IF STUDENT IS 18 OR OLDER, GO TO THAT SECTION ON NEXT PAGE.
IF STUDENT IS UNDER 18 ASK TO SPEAK TO PARENT AGAIN
Your child has qualified to participate in an interview session with a researcher in (INSERT
LOCATION) to discuss survey questions about their school environment. The interview is
scheduled for (INSERT DAY/DATE AND TIME). The session will last 60 minutes. The interview is
strictly for research purposes. Let me assure you that no one will attempt to sell or enroll you in
anything. $25 will be given to each participant to thank them for their time and participation and,
in addition, we will give you $25 as a thank you for your time if you will bring your child to and
from the facility. Will your child be able to attend the interview?
SELECT ONE
Yes
(CONTINUE)
No
(TRY FOR ALTERNATE TIME. IF NONE WORK, THANK AND TERMINATE.)
In order for your child to participate in the research study, you must sign and return a parental
consent form. You can either bring the signed form to the interview or you can scan it and email
it back. Without a signed form your child cannot participate in the interview and receive the $25.
May I email you a parental consent form?
SELECT ONE
Yes (RECORD EMAIL ADDRESS)
EMAIL________________________________________________
No --Please remember that you will need to sign the consent form when you arrive or your child
cannot participate in the interview and receive $25.
33
Recruitment Script for Student/Parent Cognitive Interviews – OMB# 1850-0803
IF STUDENT IS 18 OR OLDER
You have qualified to participate in an interview session with a researcher in Evanston/Chicago to
discuss survey questions about their school environment. The interview is scheduled for (INSERT
DAY/DATE AND TIME). The session will last 60 minutes. The interview is strictly for research
purposes. Let me assure you that no one will attempt to sell or enroll you in anything. $25 will
be given to each participant, plus $25 to a parent or guardian if they bring you to and from the
interview. Would you able to attend the interview?
SELECT ONE
Yes
(CONTINUE)
No
(TRY FOR ALTERNATE TIME. IF NOT AVAILABLE FOR ANY TIMES, THANK AND
TERMINATE)
Thank you for speaking with me today. I just wanted to let you know that the interview is scheduled for
(INSERT DAY/DATE AND TIME). The session will last 60 minutes. The interview is strictly for research
purposes. Let me assure you that no one will attempt to sell or enroll you in anything. Your child will
receive $25. If you end up bringing you child to and from the interview, we will also give you $25 as a
thank you for your time and effort.
34
Recruitment Script for Teachers Cognitive Interviews – OMB# 1850-0803
School Climate
Teacher Screener
Screener
ID________________
Date______________
Recruiter___________
Hello. My name is [XXXXXX] and I work for Research Support Services. We are
___
conducting a brief but important study about school climate. Your participation is
voluntary, and you will receive no sales pitch or follow-up calls based on your
participation. I’d like to tell you a little bit about the study first; is now a good time? (YES or NO and call
back later)
We are looking for school staff members to help us understand how they interpret school climate survey
questions. These survey questions are part of a new suite of school climate surveys that are being
developed by the U.S. Department of Education. The objective of the current study is to assess the clarity
of the survey questions, and we need help from people who work at schools to do that. The information
gained from the study will be used for potential revisions to the survey questions. Research Support
Services (RSS) will be conducting the interviews. I’d like to ask you a few questions to see if you qualify
for the study.
i. What is your name? __________________
ii. Are you male or female?
a. Male
b. Female
1. Do you currently work at an elementary, middle, or high school?
a. Yes
b. No [THANK AND TERMINATE]
2. What type of school do you work at; is it an elementary school, a middle school, or a high
school? (CHECK ALL THAT APPLY)
a. Elementary school only (no grades above 5) [THANK AND TERMINATE]
b. Middle school / junior high (roughly grades between 5-9) [RECRUIT MIX OF
MIDDLE/HIGH; BE SURE TO HAVE AT LEAST SOME MS ONLY AND SOME HS ONLY]
c. High school (ROUGHLY GRADES BETWEEN 9-12)
3. Is this a (read school types):
a. Public school, [RECRUIT 2/3 PUBLIC, 1/3 OTHERS]
b. Charter school, or
c. Private school?
35
Recruitment Script for Teachers Cognitive Interviews – OMB# 1850-0803
4. ONLY ASK IF NOT ALREADY CLEAR WHICH CATEGORY RESPONDENT FALLS INTO: What is your title
at this school?
1. Teacher for core subjects (e.g., math, science, English, foreign language, social
studies, history)
[RECRUIT A MIX OF JOB TYPES]
2. Teacher for non-core subjects (e.g., band, art, computer, music, gym)
3. Instructional staff (e.g., NOT A CLASSROOM TEACHER - teacher’s aide, instructional
coach, department head)
5. Would your school location be considered (READ RESPONSE OPTIONS):
a. Urban, [RECRUIT HALF URBAN; HALF SUBURBAN/RURAL]
b. Suburban, or
c. Rural?
d. DON’T KNOW (SPECIFY LOCATION)
______________________________________
6. Including this school year, how many years have you been working as a [title]? [IF NEEDED: at
any school, not limited to current school]
[RECRUIT A MIX OF EXPERIENCE LEVELS]
a. 2 years or less
b. 3 to 6 years
c. 7 years or more
The next few questions are for classification purposes only.
7. Are you Hispanic or Latino?
a. Yes
[RECRUIT A MIX OF YES/NO]
b. No
c. Prefer not to answer
8. Which of the following categories best describes your ethnic or racial background? (You may
select one or more races.)
[RECRUIT A MIX OF RACE/ETHNICITY BASED ON RESPONSES TO Q7/8]
a. American Indian or Alaska Native
b. Asian
c. Black or African American
d. Native Hawaiian or other Pacific Islander
e. White
f. Prefer not to answer
36
Recruitment Script for Teachers Cognitive Interviews – OMB# 1850-0803
9. What is your primary position at this school?
a. Teacher / instructional staff (e.g., teacher’s aide) [RECRUIT X]
b. Principal [RECRUIT FOR PRINCIPAL (RECRUIT X TOTAL) ; IF ALREADY HAVE ENOUGH
OF THESE RECRUITED, THANK AND TERMINATE]
c. Non-instructional staff (e.g., school counselor, lunchroom staff, nurse, janitor, bus
driver, administrative support)
[RECRUIT X; IF ALREADY HAVE ENOUGH OF
THESE RECRUITED, THANK AND TERMINATE]
d. Other (specify) __________________________ (IF TEACHING STAFF OR NONINSTRUCTIONAL STAFF, RECRUIT FOR COG LAB, IF NEEDED. OTHERWISE THANK AND
TERMINATE.)
INVITATION
You qualify to participate in an interview session with a researcher in (INSERT LOCATION) to discuss new
school climate survey questions). The interview is scheduled for (INSERT DAY/DATE AND TIME).
The session will last 60 minutes. The interview is strictly for research purposes. Let me assure you that
no one will attempt to sell or enroll you in anything. $40 will be given to each participant.
37
Recruitment Script for Non-instructional Staff Cognitive Interviews – OMB# 1850-0803
School Climate
Non-instructional Staff Screener
Screener
ID_________________
Date_______________
Hello. My name is [XXXXXX] and I work for Research Support Services. We are
conducting a brief but important study about school climate. Your participation is
voluntary, and you will receive no sales pitch or follow-up calls based on your
participation. I’d like to tell you a little bit about the study first; is now a good time?
(YES OR NO AND CALL BACK LATER)
Recruiter____________
__
We are looking for school staff members to help us understand how they interpret school climate survey
questions. These survey questions are part of a new suite of school climate surveys that are being
developed by the U.S. Department of Education. The objective of the current study is to assess the clarity
of the survey questions, and we need help from people who work at schools to do that. The information
gained from the study will be used for potential revisions to the survey questions. Research Support
Services (RSS) will be conducting the interviews. I’d like to ask you a few questions to see if you qualify
for the study.
i. What is your name? __________________
ii. Are you male or female?
a. Male
b. Female
1. Do you currently work at an elementary, middle, or high school?
a. Yes
b. No [THANK AND TERMINATE]
2. What type of school do you work at; is it an elementary school, a middle school, or a high
school? (CHECK ALL THAT APPLY)
a. Elementary school only (no grades above 5) [THANK AND TERMINATE]
b. Middle school / junior high (roughly grades between 5-9) [RECRUIT MIX OF
MIDDLE/HIGH; BE SURE TO HAVE AT LEAST SOME MS ONLY AND SOME HS ONLY]
c. High school (ROUGHLY GRADES BETWEEN 9-12)
3. Is this a (read school types):
a. Public school,
b. Charter school, or
c. Private school?
[RECRUIT 2/3 PUBLIC, 1/3 OTHERS]
38
Recruitment Script for Non-instructional Staff Cognitive Interviews – OMB# 1850-0803
4. ONLY ASK
school?
a.
b.
c.
d.
e.
f.
g.
h.
i.
IF NOT ALREADY CLEAR WHICH CATEGORY R FALLS INTO:
[RECRUIT A MIX OF JOB TYPES]
What is your title at this
School counselor
School nurse
Librarian
Administrative support staff (e.g., secretary)
Janitorial staff
Lunchroom/kitchen staff
Bus driver
Sports coach
Other
5. Would your school location be considered (read response options):
a. Urban,
[RECRUIT HALF URBAN; HALF SUBURBAN/RURAL]
b. Suburban, or
c. Rural?[DON’T READ]: Don’t know (Specify
location)______________________________________
6. Including this school year, how many years have you been working as a [title]? [IF NEEDED: at
any school, not limited to current school]
[RECRUIT A MIX OF EXPERIENCE LEVELS]
a. 2 years or less
b. 3 to 6 years
c. 7 years or more
7. Are you Hispanic?
a. Yes
b. No
c. Prefer not to answer
8. Which of the following categories best describes your ethnic or racial background? (You may
select one or more races.) [RECRUIT A MIX OF RACE/ETHNICITY BASED ON RESPONSES TO Q7/8]
a.
b.
c.
d.
e.
f.
American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White
Prefer not to answer
9. What is your primary position at this school?
39
Recruitment Script for Non-instructional Staff Cognitive Interviews – OMB# 1850-0803
a. Non-instructional staff (e.g., school counselor, lunchroom staff, nurse, janitor, bus
driver, administrative support) [RECRUIT X]
b. Principal [RECRUIT FOR PRINCIPAL (RECRUIT X TOTAL) ; IF ALREADY HAVE ENOUGH
OF THESE RECRUITED, THANK AND TERMINATE]
c. Teacher / instructional staff (e.g., teacher’s aide) [RECRUIT FOR TEACHING STAFF
(RECRUIT X TOTAL); IF ALREADY HAVE ENOUGH OF THESE RECRUITED, THANK AND
TERMINATE]
d. Other (SPECIFY) __________________________ (IF TEACHING STAFF OR
NONINSTRUCTIONAL STAFF, RECRUIT FOR COG LAB, IF NEEDED. OTHERWISE THANK
AND TERMINATE.)
INVITATION
You qualify to participate in an interview session with a researcher in (INSERT LOCATION) to discuss new
school climate survey questions). The interview is scheduled for (INSERT DAY/DATE AND TIME). The
session will last 60 minutes. The interview is strictly for research purposes. Let me assure you that no
one will attempt to sell or enroll you in anything. $40 will be given to each participant.
40
Recruitment Script for Principal’s Cognitive Interviews – OMB# 1850-0803
School Climate
Principals Screener
Screener
ID_________________
Date_______________
Hello. My name is [XXXXXX] and I work for Research Support Services. We are
conducting a brief but important study about school climate. Your participation is
voluntary, and you will receive no sales pitch or follow-up calls based on your
participation. I’d like to tell you a little bit about the study first; is now a good time?
(YES OR NO AND CALL BACK LATER)
Recruiter____________
__
We are looking for school staff members to help us understand how they interpret school climate survey
questions. These survey questions are part of a new suite of school climate surveys that are being
developed by the U.S. Department of Education. The objective of the current study is to assess the clarity
of the survey questions, and we need help from people who work at schools to do that. The information
gained from the study will be used for potential revisions to the survey questions. Research Support
Services (RSS) will be conducting the interviews. I’d like to ask you a few questions to see if you qualify
for the study.
i. What is your name? __________________
ii. Are you male or female?
a. Male
b. Female
1. Do you currently work at an elementary, middle, or high school?
a. Yes
b. No [THANK AND TERMINATE]
2. What type of school do you work at; is it an elementary school, a middle school, or a high school?
(check all that apply)
a. Elementary school only (no grades above 5) [THANK AND TERMINATE]
b. Middle school / junior high
(roughly grades between 5-9) [RECRUIT MIX OF
MIDDLE/HIGH; BE SURE TO HAVE AT LEAST SOME MS ONLY AND SOME HS ONLY]
c. High school
[ROUGHLY GRADES BETWEEN 9-12]
3. Is this a (read school types):
a. Public school,
b. Charter school, or
c. Private school?
[RECRUIT 2/3 PUBLIC, 1/3 OTHERS]
41
Recruitment Script for Principal’s Cognitive Interviews – OMB# 1850-0803
4. Would your school location be considered (read response options):
a. Urban,
[RECRUIT HALF URBAN; HALF SUBURBAN/RURAL]
b. Suburban, or
c. Rural? [DON’T READ]:
d. Don’t know (Specify location) ______________________________
5. What is your primary position at this school?
a. Principal [RECRUIT X]
b. Teacher / instructional staff (e.g., teacher’s aide) [RECRUIT FOR TEACHING STAFF
(RECRUIT X TOTAL); IF ALREADY HAVE ENOUGH OF THESE RECRUITED, THANK AND
TERMINATE]
c. Non-instructional staff (e.g., school counselor, lunchroom staff, nurse, janitor, bus driver,
administrative support)[RECRUIT FOR NON-INSTRUCTIONAL STAFF (RECRUIT X TOTAL); IF
ALREADY HAVE ENOUGH OF THESE RECRUITED, THANK AND TERMINATE]
d. Other (specify) __________________________ (IF TEACHING STAFF OR
NONINSTRUCTIONAL STAFF, RECRUIT FOR COG LAB, IF NEEDED. OTHERWISE THANK AND
TERMINATE.)
6. Including this school year, how many years have you been working as a principal? [IF NEEDED:
at any school, not limited to current school]
a. 2 years or less
[RECRUIT A MIX OF EXPERIENCE LEVELS]
b. 3 to 6 years
c. 7 years or more
7. Are you Hispanic or Latino?
a. Yes
b. No
c. Prefer not to answer
8. Which of the following categories best describes your ethnic or racial background? (You may
select one or more races.) [RECRUIT A MIX OF RACE/ETHNICITY BASED ON RESPONSES TO Q6/7]
a. American Indian or Alaska Native
b. Asian
c. Black or African American
d. Native Hawaiian or other Pacific Islander
e. White
f. Prefer not to answer
INVITATION
You qualify to participate in an interview session with a researcher in (INSERT LOCATION) to discuss new
school climate survey questions. The interview is scheduled for (INSERT DAY/DATE AND TIME). The
session will last 60 minutes. The interview is strictly for research purposes. Let me assure you that no
one will attempt to sell or enroll you in anything. $40 will be given to each participant.
42
Appendix C:
Consent/Assent Forms
43
Informed Consent – Parents/Legal Guardians of Students
School Climate Study
OMB #1850-0803 v.145
Dear Parent/Guardian,
Research Support Services (RSS) is conducting a study on behalf of the National Center for Education
Statistics, part of the U.S. Department of Education. For the study, the ED School Climate Surveys (EDSCLS),
we are seeking students between grades 5-12 to participate in an effort to gather information that will
improve the survey. For this study, an RSS staff member will meet students individually for up to 60 minutes.
Students will be given $25 to thank them for their time and participation. Parents/Guardians will be given
$25 in the event they transport their child to and from the study location.
Students will be asked to think aloud while they read student questions from the EDSCLS. Additionally, the
interviewer (RSS staff member) will ask targeted questions to gain a better understanding of student
comprehension of the survey questions. Student responses will be audio-recorded and the information
collected during this study will be for research purposes only. Your child’s participation is voluntary, and
he/she may request to stop at any time during the session.
Before your child is eligible to participate, this consent form is required. Additionally, parents are responsible
for the transportation of his/her child to and from the interview location.
If you have further questions, please contact Alisú Schoua-Glusberg at 847-864-5677, or via
[email protected].
Thank you,
Alisú Schoua-Glusberg
Research Support Services
847-864-5677
[email protected]
44
Informed Consent Form for Parents/Legal Guardians of Students
Student Name (Please Print): ____________________________________
Parent/Guardian Name (Please Print): _____________________________________
I am the parent/guardian of the student identified above. I have read the description of the ED School
Climate Surveys (EDSCLS) study and associated activities (see reverse); I hereby give my permission for my
child named above to participate in the study sessions. I understand that my child’s session will be audiorecorded. All responses may be used only for research purposes and may not be disclosed or used, in
identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002
(ESRA 2002) 20 U.S.C., § 9573].
I understand that my child’s participation is voluntary, and that he/she may stop participating at any time
during the session. I understand that I am responsible for my child’s transportation to and from the interview
location. If I have further questions and/or concerns about my child’s participation, I may contact Alisú
Schoua-Glusberg at 847-864-5677, or [email protected].
_______________________________________
__________________
Parent/Guardian Signature
Date
Please bring this to the scheduled study session.
45
Student Assent form
School Climate Study
OMB #1850-0803 v.145
I understand the research is about a new survey that asks about my opinions of the school that I currently
attend. I understand that I’m participating voluntarily, that the session will be audio-recorded, that this is a
one-time activity, and that my answers may be used only for research purposes and may not be disclosed, or
used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573).
I have read this information (or had the information read to me) I have had my questions answered and know
that I can ask questions later if I have them.
I agree to take part in the research.
OR
I do not wish to take part in the research and I have not signed the assent below.___________(initialled by
child/minor)
Only if child assents:
Print name of child ___________________
Signature of child: ____________________
Date:________________
day/month/year
Statement by the researcher/person taking consent
I have accurately read out the information sheet to the potential participant, and to the best of my ability made
sure that the child understands the cognitive interview process.
I confirm that the child was given an opportunity to ask questions about the study, and all the questions asked
by him/her have been answered correctly and to the best of my ability. I confirm that the individual has not
been coerced into giving consent, and the consent has been given freely and voluntarily.
A copy of this assent form has been provided to the participant.
Print Name of Researcher/person taking the assent________________________
Signature of Researcher /person taking the assent __________________________
Date ___________________________
Day/month/year
46
INFORMED CONSENT – TEACHER
OMB #1850-0803 v.145
Dear Teacher,
Research Support Services (RSS) is conducting a study on behalf of the National Center for Education
Statistics, part of the U.S. Department of Education. For the study, the ED School Climate Surveys (EDSCLS),
we are seeking teachers of students between grades 5-12 to participate in an effort to gather information
that will improve the survey. For this study, an RSS staff member will meet with teachers individually for up
to 60 minutes. Participants will be given $40 to thank them for their time and participation.
Teachers will be asked to think aloud while they read questions from the EDSCLS. Additionally, the
interviewer (RSS staff member) will ask targeted questions to gain a better understanding of teacher
comprehension of the survey questions. Participant responses will be audio-recorded and the information
collected during this study will be for research purposes only. Your participation is voluntary, and you may
request to stop at any time during the session.
Before you are eligible to participate, this consent form is required.
Thank you,
Alisú Schoua-Glusberg
Research Support Services
847-864-5677
[email protected]
47
Informed Consent Form - Teacher
Teacher Name (Please Print): _____________________________________
I have read the description of the ED School Climate Surveys (EDSCLS) study and associated activities (see
reverse); all of my questions have been answered. I understand that my session will be audio-recorded. All
responses may be used only for research purposes and may not be disclosed or used, in identifiable form, for
any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., §
9573]. I understand that my participation is voluntary, and that I may stop participating at any time during
the session. I agree to participate in this study.
_______________________________________
__________________
Teacher Signature
Date
Please bring this to the scheduled study session.
48
Informed Consent – Non-Instructional Staff
School Climate Study
OMB #1850-0803 v.145
Dear Non-Instructional Staff Member,
Research Support Services (RSS) is conducting a study on behalf of the National Center for Education
Statistics, part of the U.S. Department of Education. For the study, the ED School Climate Surveys (EDSCLS),
we are seeking non-instructional staff members in schools of students between grades 5-12 to participate in
an effort to gather information that will improve the survey. For this study, an RSS staff member will meet
with participants individually for up to 60 minutes. Participants will be given $40 to thank them for their time
and participation.
Participants will be asked to think aloud while they read questions from the EDSCLS. Additionally, the
interviewer (RSS staff member) will ask targeted questions to gain a better understanding of noninstructional staff comprehension of the survey questions. Participant responses will be audio-recorded and
the information collected during this study will be for research purposes only. Your participation is voluntary,
and you may request to stop at any time during the session.
Before you are eligible to participate, this consent form is required.
Thank you,
Alisú Schoua-Glusberg
Research Support Services
847-864-5677
[email protected]
49
Informed Consent Form – Non-Instructional Staff
Non-Instructional Staff Name (Please Print): _____________________________________
I have read the description of the ED School Climate Surveys (EDSCLS) study and associated activities; all of
my questions have been answered. I understand that my session will be audio-recorded. All responses may
be used only for research purposes and may not be disclosed or used, in identifiable form, for any other
purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. I
understand that my participation is voluntary, and that I may stop participating at any time during the
session. I agree to participate in this study.
_______________________________________
__________________
Non-Instructional Staff Signature
Date
Please bring this to the scheduled study session.
50
Informed Consent – Principals
School Climate Study
OMB #1850-0803 v.145
Dear Principal,
Research Support Services (RSS) is conducting a study on behalf of the National Center for Education
Statistics, part of the U.S. Department of Education. For the study, the ED School Climate Surveys (EDSCLS),
we are seeking principals of students between grades 5-12 to participate in an effort to gather information
that will improve the survey. For this study, an RSS staff member will meet with staff individually for up to 60
minutes. Participants will be given $40 to thank them for their time and participation.
Participants will be asked to think aloud while they read questions from the EDSCLS. Additionally, the
interviewer (RSS staff member) will ask targeted questions to gain a better understanding of principal
comprehension of the survey questions. Participant responses will be audio-recorded and the information
collected during this study will be for research purposes only. Your participation is voluntary, and you may
request to stop at any time during the session.
Before you are eligible to participate, this consent form is required.
Thank you,
Alisú Schoua-Glusberg
Research Support Services
847-864-5677
[email protected]
51
Informed Consent Form – Principals
Principal Name (Please Print): _____________________________________
I have read the description of the ED School Climate Surveys (EDSCLS) study and associated activities (see
reverse); all of my questions have been answered. I understand that my session will be audio-recorded. All
responses may be used only for research purposes and may not be disclosed or used, in identifiable form, for
any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., §
9573]. I understand that my participation is voluntary, and that I may stop participating at any time during
the session. I agree to participate in this study.
_______________________________________
__________________
Principal Signature
Date
Please bring this to the scheduled study session
52
Appendix D:
Questionnaires
53
School Climate Surveys
Student questionnaire
OMB #1850-0803 v.145
How strongly do you agree or disagree with the following statements about this school?
Mark One Response.
13. It is easy to talk with teachers at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
14. My teachers care about me.
Strongly Agree
Agree
Disagree
Strongly Disagree
15. At this school, there is a teacher or some other adult who students can go to if they need
help because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
16. My teachers make me feel good about myself.
Strongly Agree
Agree
Disagree
Strongly Disagree
30. I feel like I am part of this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
54
31. I feel socially accepted.
Strongly Agree
Agree
Disagree
Strongly Disagree
32. I feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
33. I feel safe going to and from this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
34. Students at this school feel unsafe because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
35. I sometimes stay home because I don’t feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
55
School Climate Surveys
Instructional/non-instructional staff questionnaire
OMB #1850-0803 v.145
How strongly do you agree or disagree with the following statements about this school?
Mark One Response
28. I can manage almost any student behavior problem.
Strongly Agree
Agree
Disagree
Strongly Disagree
29. I feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
30. The following types of problems occur at this school often: physical conflicts among
students.
Strongly Agree
Agree
Disagree
Strongly Disagree
31. The following types of problems occur at this school often: robbery or theft.
Strongly Agree
Agree
Disagree
Strongly Disagree
32. The following types of problems occur at this school often: vandalism.
Strongly Agree
Agree
Disagree
Strongly Disagree
56
33. The following types of problems occur at this school often: student possession of
weapons.
Strongly Agree
Agree
Disagree
Strongly Disagree
34. The following types of problems occur at this school often: sexual assault or violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
35. The following types of problems occur at this school often: physical abuse of teachers.
Strongly Agree
Agree
Disagree
Strongly Disagree
36. The following types of problems occur at this school often: student verbal abuse of
teachers.
Strongly Agree
Agree
Disagree
Strongly Disagree
57
Appendix E:
Protocols
58
Case ID# _________
Interviewer Initials: ______
Date:______
Time:______
School Climate Surveys
OMB #1850-0803 v.145
STUDENT SURVEY
2015
Conducted by:
U.S. Department of Education
National Center for Education Statistics
Administered by:
Research Support Services, Inc
59
MATERIALS NEEDED FOR INTERVIEW
□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)
□ RESPONDENT QUESTIONNAIRE
□ DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?
□ CONSENT FORM (4 COPIES) AND ASSENT FORM (2 COPIES)
□ INCENTIVE
□ PAYMENT RECEIPT (2 COPIES)
□ DIGITAL RECORDER AND EXTRA BATTERIES
□ PENS AND PENCILS
STEP 1: WELCOME AND INTRODUCTION SCRIPT
Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested
content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents
instructions for the interviewer.
Hello, my name is and I work for Research Support Services. It’s nice to meet you, thanks for
coming to help us out today.
[Create small talk to build rapport with the student]:
You are here today to help us out with a new survey that we are working on with the U.S. Department of
Education. This survey asks about your opinions of the school you currently attend. A survey is just a set of
questions about your experiences. This is not a test. I will ask you to take the survey as if you were taking a
real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. I will
not be grading your work. Once you are done I will ask you questions about the survey. This will help us
make the survey better and clearer for other students just like you.
You can choose whether or not you want to participate. You may also skip any question you do not want to
answer. [We have discussed this research with your parent(s)/guardian and they know that we are also
asking you for your agreement. If you are going to participate in the research, your parent(s)/guardian also
have to agree. But if you do not wish to take part in the research, you do not have to do so, even if your
parents have agreed.] Because the information you provide is so important, I am going to be taking notes
while you are working and while we are talking [INSERT IF THERE IS AN OBSERVER: and my
colleague will also be observing our session today].
Also, this interview will be recorded so researchers can review the tapes later. Your responses and any
information that refers to you or the name of your school will not be included in our report. Your responses
will be used for research purposes only and will not be disclosed, or used for any other purpose except as
required by law. [if ask: Education Sciences Reform Act of 2002, 20 U.S. Code, Section 9573]
Remember, you are not being tested or graded; we want you to tell us what you think about the survey. Would
you like to be a part of this study? [If respondent agrees, continue, if not, end the interview.]
If at any time you want to stop, just let me know.
60
[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time
and end the interview.]
Before we continue, do you have any questions about what I just said? [Answer any questions the student
may ask.]
PROVIDE RESPONDENT WITH A COPY OF THE ASSENT FORM. ASK THE RESPONDENT TO READ
THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER
ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY
OF THE FORM WITH THE RESPONDENT.
□ SIGNED ASSENT FORM COLLECTED
□ COPY OF ASSENT FORM GIVEN TO RESPONDENT
□ IF THE RESPONDENT HAS ASSENTED TO RECORDING, START THE RECORDER.
STEP 2: THINK-ALOUD PRACTICE
While you fill out the survey I want you to think out loud about how you are answering some of the questions.
Hearing you talk about how you figure out your answers to the questions will help me understand how to make
the questions better. I am more interested in how you arrive at your answers than in the answers you give me.
To help you get started on thinking aloud, we are going to do a practice activity. I’m going to ask you a
question and ask you to think aloud as you decide on your answer.
How many windows are there in the house or apartment where you live?
Please tell me what you are thinking while you are figuring out your answer.
Probe as needed for detail: How did you arrive at that answer? What were you visualizing
when you were deciding on that answer? Can you tell me more about what you are
including in your count?
When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me
about your answers to the questions in the survey.
HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN.
61
STEP 3: COMPLETION OF THE QUESTIONNAIRE
Now I would like you to complete the questionnaire. While you are answering the questions, please think about
the school you currently attend. I’d like you to think aloud while you decide on your answer. Please read aloud
anything that you would normally read to yourself if I were not here.
Remember that I am very interested in hearing you talk about what you are thinking about as you figure out
your answers to the questions.
Probe bank (from Willis)
Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are
thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question
before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”
Say to respondent: Please answer these questions thinking about the school you are attending right
now.
National Center for Education Statistics ED School Climate Surveys (EDSCLS) – Cognitive Testing Items
Throughout the survey, "This school" means activities happening in school buildings, on school grounds, on
school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this
refers to normal school hours or to times when school activities/events were in session.
How strongly do you agree or disagree with the following statements about this school? Mark One
Response.
13. It is easy to talk with teachers at this school.
Sengrel12
Strongly Agree
Agree
Disagree
Strongly Disagree
62
PROBES
1. What is this question asking?
2. What does ‘talk with teachers’ mean in this question?
14. My teachers care about me.
Sengrel14
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBES:
1. What does this question mean to you?
15. At this school, there is a teacher or some other adult who students can go to if they need help because of
sexual assault or dating violence.
Sengrel153
Strongly Agree
Agree
Disagree
Strongly Disagree
GOALS
PROBES
1) Tell me about your answer.
ARE THEY THINKING OF THEIR CURRENT
SCHOOL?
DOES “SOME OTHER ADULT” INCLUDE
ADULTS OUTSIDE OF SCHOOL?
IS “GO TO” LITERAL OR FIGURATIVE TO
THEM?
ARE THEY INCLUDING VIOLENCE THAT
HAPPENED OUTSIDE OF SCHOOL?
2) What is this question asking you?
3) What does ‘if they need help’ mean in this question
4) What does sexual assault mean in this question?
5) What does dating violence mean in this question?
63
WHAT IS THE STUDENT’S LEVEL OF
COMFORT WITH THIS Q?
WHY DID THEY CHOOSE AGREE/DISAGREE
VS. STRONGLY AGREE/STRONGLY DISAGREE
16. My teachers make me feel good about myself.
Sengrel17
Strongly Agree
Agree
Disagree
Strongly Disagree
30. I feel like I am part of this school.
Ssafemo56
Strongly Agree
Agree
Disagree
Strongly Disagree
31. I feel socially accepted.
Ssafemo57
Strongly Agree
Agree
Disagree
Strongly Disagree
32. I feel safe at this school.
Strongly Agree
Agree
Disagree
Strongly Disagree
Ssafpsaf59
PROBES:
1. What is this question asking?
2. What does ‘safe’ mean in this question?
33. I feel safe going to and from this school.
Ssafpsaf60
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBE:
1. What does to and from school mean to you?
64
34. Students at this school feel unsafe because of sexual assault or dating violence.
Strongly Agree
Agree
Disagree
Strongly Disagree
Ssafpsaf154
GOALS
PROBES
1. Tell me more about your answer?
ARE THEY THINKING OF BOTH SEXUAL
ASSAULT AND DATING VIOLENCE?
HOW DID THEY DECIDE ON THEIR
RESPONSE?
WHAT DOES “UNSAFE” MEAN TO THEM?
WHY DID THEY CHOOSE AGREE/DISAGREE
VS. STRONGLY AGREE/STRONGLY DISAGREE
2. What is this question asking?
3. You said students (do/do not) feel unsafe in your school. Tell me more about that. What does unsafe mean in this
question?
4. You answered XX tell me why you chose that answer instead of [one below]
5. And tell me about how you chose that answer instead of [one above]
35. I sometimes stay home because I don’t feel safe at this school.
Ssafpsaf63
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBE
1. What is this question asking?
STEP 4. THANK YOU FOR PARTICIPATING
We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this
survey better.
INTERVIEWER INSTRUCTIONS: COLLECT MATERIALS AND ESCORT STUDENT OUT OF THE INTERVIEWING ROOM
AND BACK TO HIS OR HER PARENT. PLEASE GIVE THE STUDENT AN INCENTIVE [$25], THE PARENT AN INCENTIVE
[$25] IF THEY BROUGHT THEIR CHILD TO AND FROM THE INTERVIEW, HAVE THEM SIGN RECEIPTS, AND THANK
THEM FOR THEIR TIME AND EFFORT.
65
Case ID#________
Interviewer Initials ______
Date:________
Time:________
School Climate Surveys
OMB #1850-0803 v.145
INSTRUCTIONAL/NON-INSTRUCTIONAL STAFF SURVEY
2015
Conducted by:
U.S. Department of Education
National Center for Education Statistics
Administered by:
Research Support Services, Inc.
2 or less
Years Working in Current Title
3-6
7 or more
Core
Staff Title _______________
Non-Core
Instructional Staff
Non-Instructional Staff
MATERIALS NEEDED FOR INTERVIEW
□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)
□ RESPONDENT QUESTIONNAIRE
□ DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?
□ CONSENT FORM (TWO COPIES)
□ INCENTIVE
□ PAYMENT RECEIPT
□ DIGITAL RECORDER AND EXTRA BATTERIES
□ PENS AND PENCILS
66
STEP 1: WELCOME AND INTRODUCTION SCRIPT
Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested
content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents
instructions for the interviewer.
Hello, my name is and I work for Research Support Services, Inc. It’s nice to meet you, thank
you for coming to help us out today.
[Create small talk to build rapport with the teacher/Admin]:
You are here today to help us out with a new survey that we are working on with the U.S. Department of
Education. This survey asks about your opinions of the school where you currently work. I will ask you to
take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do
not know some of the answers. As you are answering the survey questions I will ask you about them. I will
also stop you once in a while to ask you questions. This will help us make the survey better and clearer for
other teachers and school instructional staff just like you.
Because the information you provide is so important, I am going to be taking notes while you are working
and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be
observing our session today].
Also, this interview will be recorded so researchers can review the tapes later. Your responses and any
information that refers to you or the name of your school will not be included in our report. Your responses
will be used for research purposes only and will not be disclosed, or used for any other purpose except as
required by law. [if ask: Education Sciences Reform Act of 2002, 20 U.S. Code, Section 9573]
Remember, in this study we want you to tell us what you think about the survey. Would you like to be a part of
this study? [If respondent agrees, continue, if not, end the interview.]
If at any time you want to stop, just let me know.
[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time
and end the interview.]
Before we continue, do you have any questions about what I just said? [Answer any questions the participant
may ask.]
PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE
RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT
PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM.
LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.
□ SIGNED CONSENT FORM COLLECTED
□ COPY OF CONSENT FORM GIVEN TO RESPONDENT
□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE
RECORDER.
67
STEP 2: THINK-ALOUD PRACTICE
While you fill out the survey I want you to think out loud about how you are answering some of the questions.
Hearing you talk about how you figure out your answers to the questions will help me understand how to make
the questions better. I am more interested in how you arrive at your answers than in the answers you give me.
To help you get started on thinking aloud, we are going to do a practice activity. I’m going to ask you a
question and ask you to think aloud as you decide on your answer.
How many windows are there in the house or apartment where you live?
Please tell me what you are thinking while you are figuring out your answer.
Probe as needed for detail: How did you arrive at that answer? What were you visualizing
when you were deciding on that answer? Can you tell me more about what you are
including in your count?
When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me
about your answers to the questions in the survey.
HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN.
STEP 3: COMPLETION OF THE QUESTIONNAIRE
Now I would like you to complete the questionnaire. While you are answering the questions, please think about
the school where you currently work. I’d like you to think aloud while you decide on your answer. Please read
aloud anything you would have read to yourself if I were not here.
Remember that I am very interested in hearing you talk about what you are thinking about as you figure out
your answers to the questions.
Probe bank (from Willis)
Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before
answering this question]: “Tell me what just happened. Tell me about what you were just doing.”
68
How strongly do you agree or disagree with the following statements about this school? Mark One Response
28. I can manage almost any student behavior problem.
Isafemo58
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBES:
1. What is this question asking?
2. When you answered this question, what kinds of 'student behavior problem' did you think of?
29. I feel safe at this school.
Isafpsaf59
Strongly Agree
Agree
Disagree
Strongly Disagree
30. The following types of problems occur at this school often: physical conflicts among students.
Isafpsaf60
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBES:
1. What is this question asking?
2. IF AGREE OR STRONGLY AGREE: Tell me about the kinds of physical conflicts that occur.
IF DISAGREE OR STRONGLY DISAGREE: What do you think is meant in this question by physical conflicts?
69
31. The following types of problems occur at this school often: robbery or theft.
Isafpsaf61
Strongly Agree
Agree
Disagree
Strongly Disagree
32. The following types of problems occur at this school often: vandalism.
Isafpsaf62
Strongly Agree
Agree
Disagree
Strongly Disagree
33. The following types of problems occur at this school often: student possession of weapons.
Isafpsaf64
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBE:
1. Tell me about your answer
2. What do you think they mean here by 'often'?
34. The following types of problems occur at this school often: sexual assault or violence.
Isafpsaf143
Strongly Agree
GOALS
Agree
Disagree
Strongly Disagree
DO THEY UNDERSTAND THE Q?
ARE THEY THINKING OF SEXUAL VIOLENCE
AS OR SEXUAL ASSAULT AND VIOLENCE
SEPARATELY?
PROBES:
1) Tell me about your answer.
2) What is this question asking you?
3) What does ‘sexual assault or violence’ mean in this question?
4) What do you think they mean by ‘often’ here?
70
ARE THEY THINKING OF THEIR SCHOOL?
WHY DID THEY CHOOSE AGREE/DISAGREE
VS. STRONGLY AGREE/STRONGLY DISAGREE
35. The following types of problems occur at this school often: physical abuse of teachers.
Isafpsaf66
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBES:
1) Tell me about your answer.
2) What does physical abuse mean in this question?
36. The following types of problems occur at this school often: student verbal abuse of teachers.
Isafpsaf67
Strongly Agree
Agree
Disagree
Strongly Disagree
PROBES:
1) Tell me about your answer.
2. What would you consider often for this type of occurrence?
STEP 4. THANK YOU FOR PARTICIPATING
We are finished. Thank you for helping us. The information you’ve provided will be very helpful for
making this survey better.
INTERVIEWER INSTRUCTIONS: COLLECT MATERIALS. PLEASE GIVE THE RESPONDENT AN INCENTIVE, AND
HAVE THEM SIGN RECEIPT [$40].
71
| File Type | application/pdf |
| Author | Jessie Engel |
| File Modified | 2016-01-13 |
| File Created | 2016-01-13 |