School Survey on Crime and Safety (SSOCS) 2018 Cognitive Interviews 2016

National Center for Education Statistics (NCES)

School Survey on Crime and Safety (SSOCS) 2018 Cognitive Interviews 2016

Appendix F

Cognitive Laboratory Protocol

OMB #1850-0803 v.171

September 2016

Cognitive Laboratory: Protocol for SSOCS Survey

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ SSOCS SURVEY INTRODUCTION MATERIALS (COVER LETTER, ADVANCE LETTER, BROCHURE)

□ RESPONDENT QUESTIONNAIRE

□ DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?

□ THINK ALOUD PRACTICE PAGE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PENCILS

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is <your name> and I work for the American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the principal]:

You are here today to help us out with a survey that we are working on with the U.S. Department of Education. This survey asks about your opinions of the school where you work: [school name]. A survey is a set of questions about your experiences. I will ask you to take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will stop you once in a while to ask you questions. This will help us make the survey better and clearer for other principals and school staff just like you.

Because the information you provide is so important, I am going to be taking notes while you are working and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today].

Also, this interview will be recorded so researchers can review the tapes later. Any information that refers to you or the name of your school will not be included in our report. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573).

As a thank you for your time and effort, you will receive a [INSERT GIFT CARD AMOUNT ($75 IN-PERSON INTERVIEW/$50 REMOTE INTERVIEW)] gift card at the end of today’s interview.

Remember, you are not being tested; we want you to tell us what you think about the survey. Would you like to be a part of this study? [If respondent agrees, continue. If not, end the interview.]

If at any time you want to stop, just let me know. [If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the participant may ask.]

PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.

□ SIGNED CONSENT FORM COLLECTED

□ COPY OF CONSENT FORM GIVEN TO RESPONDENT

□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to think out loud about how you are answering the questions. Hearing you talk about how you figure out your answers to the questions will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give me. To help you get started on thinking aloud, we are going to do a practice activity.

IF IN-PERSON: I’m going to give you a piece of paper with a question in it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer. [Give respondent handout with think aloud practice.]

IF REMOTE: I’m going to ask you a question, and I would like you to think aloud as you decide on your answer.

QUESTION: How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

STEP 3: REVIEW SSOCS MATERIALS

IF IN-PERSON: Before we begin the questionnaire, I’d like to first review the materials that you receive either in preparation of the survey or with the survey. [Hand respondent the SSOCS Survey Introduction Materials (including the advanced letter, cover letter, and brochure.)]

IF REMOTE: Before we begin the questionnaire, I’d like to first review the materials that you receive either in preparation of the survey or with the survey. Please open the attachment from the email I sent you earlier that is called “SSOCS Survey Introduction Materials”.

[Referring to the Advanced Letter]

1. In your own words, can you tell me what this letter is for?

2. Can you tell me who this letter is from?

3. In your own words, can you tell me what this survey is about?

4. In your own words, can you tell me what the information gathered from this survey is going to be used for?

5. How would you improve this letter?

6. After you received this letter, would you take any steps to prepare for the SSOCS survey? If yes, what would those steps be?

[Referring to the Brochure]

1. In your own words, can you tell me what this brochure is telling you?

2. Is there any additional information that you would want provided in the brochure?

3. Is receiving this brochure helpful or not helpful? (PROBE: Why is that?)

4. Do you prefer to receive the brochure in the mail or is there a better way to access this information?

[Referring to the Physical Envelope/Pen]

1. What would be your response if you received this FedEx envelope at your school? Why?

2. What would increase the likelihood that you would open the envelope?

3. Would your approach to this survey change if it were delivered by USPS or priority mail? Why?

4. Would the inclusion of a small gift, such as a free pen, increase the likelihood that you would respond to the survey?

[Referring to all the materials together]

1. Based on these materials, are there any reasons why you would choose not to complete this survey?

2. Are there materials or information not included that would increase the likelihood that you would respond to the survey? What would they be?

3. Do you have anything else you would like to tell me about these materials that you haven’t had a chance to mention?

STEP 4: COMPLETION OF THE QUESTIONNAIRE

IF IN-PERSON: Now I would like you to complete the questionnaire. [Hand respondent a copy of the questionnaire and a pen.]

IF REMOTE: Now I would like you to complete the questionnaire. Please open the attachment “SSOCS 2016 Questionnaire.” If you have printed the questionnaire out, please pull it out.

While you are answering the questions, please think about the school you currently work at: [school name]. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would have read to yourself if I were not here.

Remember that I am very interested in hearing you talk about what you are thinking about as you figure out your answers to the questions.

General Probes – These should only be used when the R shows signs of confusion to a question that does not have a specific probe
Problem Items	What is this question asking, in your own words?
	What were you thinking when you answered this question?
Problem Instructions	Tell me what this introduction/instruction is telling you.
Problems with Clarity	Can you tell me in your own words what this question is asking?
	What does this word [term] mean to you? [in reference to a particular word of technical term]
	Tell me what you were thinking when I asked you about this [topic, time period, degree, certification, etc…]
Skip and Marking Issues	I noticed you skipped this/checked this. How did you choose this answer and not this other option?
	Can you tell me how you chose this category? How easy or hard was it to find your answer on that list

START THE QUESTIONNAIRE

SECTION: School Practices and Programs
STOP Respondent after Question 4 – I’d like you to stop here
ALL: Were there any questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.
1. During the 2015–16 school year, was it a practice of your school to do the following? If your school changed its practices during the school year, please answer regarding your most recent practice. Check “Yes” or “No” on each line.
a. Perform one or more random sweeps for contraband (e.g., drugs or weapons*), including dog sniffs	Can you tell me in your own words what this question is asking? What is a sweep? What does this term mean for you? Can you describe this for me?
b. Prohibit non-academic use of cell phones or smartphones during school hours	What were you thinking when you answered this question? Does your school allow for the use of cell phones or smartphones in certain situations or for specific purposes? Can you elaborate on these situations or purposes?
Glossary term referenced: Weapon – any instrument or object used with the intent to threaten, injure, or kill. This includes look-alikes if they are used to threaten others.
2. Does your school have a written plan that describes procedures to be performed in the following scenarios?
General	What is this question asking, in your own words?
c. Accidents (e.g., school bus or other vehicle accidents; injury during a school-sponsored activity or trip)	What were you thinking when you answered this question? Are there other types of accidents beyond the examples here that are included in your school plans?
3. During the 2015–16 school year, did your school have any formal programs intended to prevent or reduce violence* that included the following components for students? If a program has multiple components, answer "Yes" for each that applies. Check "Yes" or "No" on each line.
General	Can you tell me in your own words what this question is asking? What would a program need to include for you to consider it to have the intention to prevent or reduce violence? How did you handle this question if a component listed here is included in a program in your school without the specific intention to prevent or reduce violence? [WHAT WE WANT TO KNOW: Try to find out if principals are marking components that are specifically in “programs intended to prevent or reduce violence” – as noted in the stem – or if they are marking components for any programs they have, regardless of if the program has a specific focus on violence prevention/reduction.]
Glossary term referenced: Violence – actual, attempted, or threatened fight or assault.
4. To the best of your knowledge, during the 2015–16 school year, were there any staff at your school who legally carried a firearm* on school property? Exclude School Resource Officers, other sworn law enforcement officers, or other security guards or personnel who carry firearms.
General	Can you tell me in your own words what this question is asking? If answered yes: Can you tell me about any additional policies or practices that staff who carry firearms must follow? [WHAT WE WANT TO KNOW: Try to find out how principals would know about policies or know if their staff carry guns and if there are particular rules the staff have to follow if they do bring a gun to school (specific training, ways of securing, etc.).]
Glossary term referenced: Firearm/explosive device – any weapon that is designed to (or may readily be converted to) expel a projectile by the action of an explosive. This includes guns, bombs, grenades, mines, rockets, missiles, pipe bombs, or similar devices designed to explode and capable of causing bodily harm or property damage.
LET THE RESPONDENT CONTINUE WITH THE SURVEY.

SECTION: School Security Staff
STOP Respondent after Question 5 – I’d like you to stop here
5. During the 2015–16 school year, did your school have one or more school-based probation officers* located on school property?
General		Can you tell me in your own words what this question is asking? Can you tell me what “probation officer” means to you in the context of this question? How did you come up with your answer? Tell me more about your answer? [WHAT WE WANT TO KNOW: Try to find out if principals know if students at their school are under probation, and if probation officers play a different role if they aren’t onsite.]
	Glossary term referenced: Probation officers – sometimes referred to as community supervision officers, supervises people who have been placed on probation instead of sent to prison. They work to ensure that the probationer is not a danger to the community and to help in their rehabilitation through frequent visits with the probationer. Probation officers write reports that detail each probationer’s treatment plan and their progress since being put on probation. Most work exclusively with either adults or juveniles.
	LET THE RESPONDENT CONTINUE WITH THE SURVEY.

Section: School Mental Health Services
STOP Respondent after Question 9 – I’d like you to stop here
6. During the 2015–16 school year, did your school have the ability to provide diagnostic assessment* (e.g. screening, psychiatric assessment) to students for mental health disorders*? Include services that were provided at school* as well as services provided through a contract the school has with an outside provider.
OBSERVE: This is a yes/no question with a skip pattern if “no” is chosen. Does respondent follow the skip pattern?
General	Can you tell me what “diagnostic assessment” means to you?
General	Can you tell me what “mental health disorder” means to you?
Glossary terms referenced: At school/at your school – activities happening in school buildings, on school grounds, on school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this refers to normal school hours or to times when school activities/events were in session. Diagnostic assessment – an evaluation conducted by a medical or mental health professional that identifies whether an individual has one or more medical and/or mental health diagnoses. This is in contrast to an educational assessment, which does not focus on clarifying a student’s diagnosis. Mental health disorders – collectively, all diagnosable mental disorders or health conditions that are characterized by alterations in thinking, mood, or behavior (or some combination thereof) associated with distress and/or impaired functioning.
7. Where were diagnostic assessment* services available to students from your school?
a. At school*, by a school-employed or contracted mental health professional*	In the context of this question, can you tell me what “mental health professional” means to you?
b. Outside of school, by a school-employed or contracted mental health professional*	In the context of this question, can you tell me what “mental health professional” means to you?
General	Does your school have a contract with mental health professionals not employed by the school to provide diagnostic assessment services to students, either at school or outside of school? If yes: Could you tell me more about these contracted relationships? Are there other ways that your school is providing diagnostics assessment services to students from your school? If yes: What are they?
Glossary terms referenced: At school/at your school – activities happening in school buildings, on school grounds, on school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this refers to normal school hours or to times when school activities/events were in session. Diagnostic assessment – an evaluation conducted by a medical or mental health professional that identifies whether an individual has one or more medical and/or mental health diagnoses. This is in contrast to an educational assessment, which does not focus on clarifying a student’s diagnosis. Mental health professionals – mental health services are provided by several different professions, each of which has its own training and areas of expertise. The types of professionals who may provide mental health services include psychiatrists, psychologists, psychiatric/mental health nurse practitioners, psychiatric/mental health nurses, clinical social workers, and professional counselors.
8. During the 2015–16 school year, did your school have the ability to provide treatment* (e.g. psychotherapy, medication) to students for mental health disorders*? Include services that were provided at school* as well as services provided through a contract the school has with an outside provider.
OBSERVE: This is a yes/no question with a skip pattern if “no” is chosen. Does respondent follow the skip pattern?
General	Can you tell me what “treatment” means to you?
General	Can you tell me what “mental health disorder” means to you?
Glossary terms referenced: At school/at your school – activities happening in school buildings, on school grounds, on school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this refers to normal school hours or to times when school activities/events were in session. Mental health disorders – collectively, all diagnosable mental disorders or health conditions that are characterized by alterations in thinking, mood, or behavior (or some combination thereof) associated with distress and/or impaired functioning. Treatment – a clinical service addressed at lessening or eliminating the symptoms of a disorder. In mental health, this may include psychotherapy, medication treatment, and/or counseling.
9. Where were treatment* services available to students from your school?
a. At school*, by a school-employed or contracted mental health professional*	In the context of this question, can you tell me what “mental health professional” means to you?
b. Outside of school, by a school-employed or contracted mental health professional*	In the context of this question, can you tell me what “mental health professional” means to you?
General	Does your school have a contract with mental health professionals not employed by the school to provide treatment services to students, either at school or outside of school? If yes: Could you tell me more about these contracted relationships? Are there other ways that your school is providing treatment services to students from your school? If yes, what are they?
Glossary terms referenced: At school/at your school – activities happening in school buildings, on school grounds, on school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this refers to normal school hours or to times when school activities/events were in session. Mental health professionals – mental health services are provided by several different professions, each of which has its own training and areas of expertise. The types of professionals who may provide mental health services include psychiatrists, psychologists, psychiatric/mental health nurse practitioners, psychiatric/mental health nurses, clinical social workers, and professional counselors. Treatment – a clinical service addressed at lessening or eliminating the symptoms of a disorder. In mental health, this may include psychotherapy, medication treatment, and/or counseling.
LET THE RESPONDENT CONTINUE WITH THE SURVEY.

Section: Staff Training
STOP Respondent after Question 10 – I’d like you to stop here
10. During the 2015–16 school year, did your school or school district provide any of the following for classroom teachers or aides?
a. Training in recognizing signs of self-harm or suicidal tendencies	Can you tell me in your own words what this question is asking?
b. Training in trauma sensitivity*	Can you tell me what “trauma sensitivity” means to you in the context of this question?
Glossary terms referenced: Trauma sensitivity – a trauma-sensitive school is one in which all students feel safe, welcomed, and supported and where addressing trauma’s impact on learning on a school-wide basis is at the center of its educational mission.
LET THE RESPONDENT CONTINUE WITH THE SURVEY.

SECTION: Number of Incidents
STOP Respondent after Question 12 – I’d like you to stop here
11. Please record the number of arrests* that occurred at your school during the 2015–16 school year. Please include all arrests that occurred at school*, regardless of whether a student or non-student was arrested.
General	How did you come up with your answer? Tell me more about your answer? Can you tell me what “arrests” means to you in the context of this question?
Glossary terms referenced: Arrest – the act of detaining in legal custody. An "arrest" is the deprivation of a person's liberty by legal authority in response to a criminal charge. At school/at your school – activities happening in school buildings, on school grounds, on school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this refers to normal school hours or to times when school activities/events were in session.
12. To the best of your knowledge, during the 2015–16 school year, have there been any incidents of sexual misconduct* between a staff member at your school and a student at your school? Report on misconduct between staff and students whether or not the incidents occurred at school or away from school.
General	How did you come up with your answer? Tell me more about your answer? Can you tell me what “sexual misconduct” means to you in the context of this question? If answered yes: About how many incidents of sexual misconduct occurred during the 2015–16 school year?
Glossary terms referenced: Sexual misconduct – any act, including, but not limited to, any verbal, nonverbal, written or electronic communication or physical activity, directed toward or with a student regardless of the age of the student that is designed to establish a romantic or sexual relationship with the student. School staff have power over students by virtue of their position, thus student-staff relationships are not equal and students cannot be consenting parties to romantic or sexual relationships.
END OF INTERVIEW

SECTION: Disciplinary Problems and Actions
STOP Respondent after Question 13 – I’d like you to stop here
ALL: Were there any questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.
13. To the best of your knowledge, how often do the following types of problems occur at your school*?
General	Can you tell me in your own words what this question is asking?
a. Student racial/ethnic tensions	What were you thinking when you answered this question?
b. Student sexual harassment* of other students	Can you tell me in your own words what this question is asking? Can you tell me what “sexual harassment” means to you?
c. Student harassment* of other students based on sexual orientation*	Can you tell me in your own words what this question is asking? Can you tell me what “harassment” means to you? Can you tell me what “sexual orientation” means to you?
d. Student harassment* of other students based on gender identity*	Can you tell me what “gender identity” means to you?
e. Student harassment* of other students based on religion	What were you thinking when you answered this question?
f. Student harassment* of other students based on disability	Can you tell me in your own words what this question is asking?
General	What comes to mind after answering these questions?
Glossary terms referenced: At school/at your school – activities happening in school buildings, on school grounds, on school buses, and at places that hold school-sponsored events or activities. Unless otherwise specified, this refers to normal school hours or to times when school activities/events were in session. Gender identity – means one’s inner sense of one’s own gender, which may or may not match the sex assigned at birth. Different people choose to express their gender identity differently. For some, gender may be expressed through, for example, dress, grooming, mannerisms, speech patterns, and social interactions. Gender expression usually ranges between masculine and feminine, and some transgender people express their gender consistent with how they identify internally, rather than in accordance with the sex they were assigned at birth. Harassment – harassing conduct may take many forms, including verbal acts and name-calling, as well as non-verbal behavior, such as graphic and written statements, or conduct that is physically threatening, harmful or humiliating. Sexual harassment – conduct that is unwelcome, sexual in nature, and denies or limits a student’s ability to participate in or benefit from a school’s education program. All students, regardless of sex or gender identity, can be victims of sexual harassment, and the harasser and the victim can be of the same sex. The conduct can be verbal, nonverbal, or physical. Sexual orientation – means one’s emotional or physical attraction to the same and/or opposite sex.
END OF INTERVIEW

AT THE END OF THE SURVEY

If this is a remote interview, please ask the respondent for his or her mailing address (so that we can send the incentive).

Name ____________________________________

Street ____________________________________

City ____________________________________

State ____________________________________

Zip ____________________________________