School Survey on Crime and Safety (SSOCS) 2018 Usability Testing

National Center for Education Statistics (NCES)

School Survey on Crime and Safety (SSOCS) 2018

Usability Testing

Appendixes K – L

Protocol and Questionnaire for User Interaction Satisfaction (QUIS)

OMB #1850-0803 v.208

September 2017

Appendix K – Protocol 2

Appendix L – Questionnaire for User Interaction Satisfaction (QUIS) 23

Appendix K – Protocol

Usability Testing Protocol for SSOCS:2018

Interview ID
Interview Date and Time
Interviewer Name
Participant Name
School Name
Staff Position

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ COPY OF PAPER QUESTIONNAIRE

□ INVITATION LETTER WITH 8-DIGIT USER ID

□ QUIS SURVEY

□ PARTICIPANTS’ COMPUTER

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ EXTRA LAPTOP AND CHARGING CORD

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] represents instructions for the interviewer.

Hello, my name is <your name >, and I work for the American Institutes for Research. It’s nice to meet you. Thanks for helping us out today.

[Create small talk to build rapport with the respondent.]

You are here today to help us test a web version of the School Survey on Crime and Safety (SSOCS). This survey asks about the school where you work: <school name>.

Today I’m interested in getting your opinions about the website that we will use to conduct the survey. I will ask you to go through the survey and use the features as if you were taking the real survey, but I will also stop you once in a while to ask you questions or ask you to complete a specific task. When I give you a task, I might also ask you to explain how you are going about the task.

This is not an evaluation of how familiar you are with the Internet or computers. There are no right or wrong answers to the survey either, and it’s ok if you do not know how to respond to some items. While the survey instructions may ask you to reference school records or databases, in today’s interview, you may estimate your answer when responding to these items. This user testing will help us make the website better and clearer for other respondents just like you. As a thank-you for your time and effort, you will receive a gift card after today’s interview.

Because the information you provide is so important, I am going to take notes while you are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today]. We’d also like to record this interview so researchers can review the tapes later. Any information that refers to you or the name of your school will not be included in our report. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Is it alright if we record the interview? [Turn on recorder if permission is given.] So that we have documentation of your consent on the recording, could you please confirm that it is alright that we record this interview?

As you complete the survey, remember, you are not being tested; all we want is to learn how well the survey works. Before we continue, do you have any questions regarding this study? [Answer any questions they may ask.]

Given the information I just provided, would you still like to participate in this study? [If for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview. If they do want to participate, then they need to sign a consent form.]

• IN-PERSON INTERVIEW:

  • If respondent emailed a completed consent form: We already have your consent form that you sent via email. Could you please confirm verbally that you have signed and returned the consent form?

  • If respondent didn’t email a completed consent form: Here is a copy of the consent form and an extra copy for your records. [Provide respondent with a copy of the consent form.] Please read the form and sign it. If you have any questions, I am happy to answer them. [Collect form from respondent.]

• REMOTE INTERVIEW:

  • If respondent emailed a completed consent form: We already have your consent form that you sent via email. Could you please confirm verbally that you have signed and returned the consent form?

  • If respondent didn’t email a completed consent form: We sent you an email with a copy of the consent form. Would you be able to sign that form and email it back to us after today’s interview? For the time being, if you consent to participate in this study, can you please type your name and “I consent” into the chat box?

Thank you. Remember that if at any time you want to stop, just let me know.

STEP 2: THINK-ALOUD PRACTICE (5 minutes)

As you move through the web survey, there may be times when I ask you to talk about what you are thinking when completing a task. Hearing you talk about how you figure out your answers to my questions or complete a task will help me understand how to make the survey better. I am more interested in how you complete the tasks than in the answers you give on the survey.

To help you get started, we are going to do a practice activity. I’m going to give you a task related to your <laptop/desktop> and ask you to talk out loud while you complete it.

Please use your internet browser to find the mailing address for the U.S. Department of Education.

Please tell me what you are thinking while you are figuring out how to find the mailing address.

[Probe as needed for detail with questions such as: Where do you plan to click first? Can you tell me more about why you clicked [xxx]? If the respondent is having trouble with the think-aloud process, illustrate for the respondent how to talk aloud when completing this task. Then ask the respondent to try again.]

Thank you; that’s the kind of detail I’m looking for when you talk to me about how you will complete the assigned tasks.

Probe Bank:

• What are you thinking about?

• Please tell me what you are thinking.

• What are you planning to do next?

• Is there anything else you can do?

• [If you pick up on a visual cue like a furrowed brow or a puzzled look]: Is there anything confusing on this page?

• [If you pick up on a visual cue that indicates confusion (e.g., going to previous pages or re-reading instructions)]: “Please tell me what you are thinking.” and/or “Please tell me about what you were just doing.”

• [In response to R actions or comments]: What does the feature mean to you?

• [In response to R actions or comments]: Please tell me what happened. Please explain to me what you were just doing.

Observations to keep in mind:

NOTE that some of these may be difficult to observe during a remote interview. If necessary, probe the respondent directly on the items above.

• Does user try to use the enter key to move to the next page?

• Does user try to use the browser back button to move backward in the survey, instead of the “Previous” button?

• How does the user respond to error messages?

• How does the user feel about the overall look of the site?

• Does the user seem to have a hard time finding the “Next” or “Previous” button?

• Does the user go back and forth between pages (follow up on why, if not obvious)?

• Does the user try to go back multiple pages, only to get confused? How do they react?

• Does the user skip any questions? Accidentally or on purpose?

• Note if any difficulty is related to content or usability (when expressed verbally).

• Note any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can they find their way out?

• Note if users are delayed in responding to an item.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Task 1: Log into the survey

Record what browser the R is using: Chrome Firefox IE Other:____________________

Record what type of device the R is using: Mac PC Other:____________________

[If this cannot be observed: Before we begin, could you tell me what browser and/or device you are using?]

IN-PERSON: [Hand respondent invitation letter (which includes the User ID)] Now it is time to complete the survey online using your preferred web browser. Please use this letter to sign in. Remember that I am very interested in hearing you talk about what you are thinking about as you figure out how to do the task. [Give respondent time to read the letter.]

REMOTE: We sent you an email that contained several documents for today’s interview. Can you please find the email and open the PDF document called “SSOCS Principal Invitation Letter”? You will need this letter to sign in and complete the survey online. Remember that I am very interested in hearing you talk about what you are thinking about as you figure out how to do the task. [Give respondent time to read the letter.]

OBSERVE if respondent has any difficulty using the invitation letter, finding their PIN, or typing in the URL.

[If the participant is having difficulty, then ask the following questions.]

Are you finding it easy or difficult to understand the instructions in the letter? Why?

Is there any information that you need from the letter in order to log into the survey?

Record the R’s PIN number: ___________________________

Did the respondent write down their PIN? (Y/N): ___________________________

Are you able to find a verification question that you will easily remember the answer to?

What type of device you would normally use to respond to a survey like this?

Task 2: Complete questionnaire

Now, please complete the questionnaire. Please think aloud as you answer the questions and remember that for today’s interview you can provide estimates for your answers. You do not need to reference school records or databases since we’re going to focus on the format of the SSOCS web survey and the process of answering questions, rather than in the answers you give. We also want to hear about question layouts that you think are awkward or difficult to interpret and resources in the tool that are difficult to find or understand.

[Instructions for interviewers]

• This symbol indicates this is a critical probe. Interviewers should make every effort to complete all critical probes.

SSOCS instrument-specific observations to keep in mind:

• If all sub-items do not appear on the screen, OBSERVE whether the respondent scrolls to find the additional items.

• For any item with a defined term – those in bold font and marked with an asterisk – the respondent is able to click on the term and a pop-up should appear with the definition. OBSERVE whether the respondent recognizes that definitions are available. OBSERVE whether the respondent clicks on an asterisked word or whether he or she goes to the definitions page. OBSERVE whether respondents have any difficulty returning to the question after accessing a definition.

OBSERVE whether respondent clicks to the “Instructions” tab before beginning the survey.

SECTION: School Practices and Programs

1. During the 2017–18 school year, was it a practice of your school to do the following?

• If your school changed its practices during the school year, please answer regarding your most recent practice.

• OBSERVE if the respondent has any difficulty with the radio buttons.

• OBSERVE whether the respondent has any difficulty locating the “Next” button.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: weapons*

2. Does your school have a written plan that describes procedures to be performed in the following scenarios?

• OBSERVE if the respondent has any reaction to the appearance of the “Previous” button.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: active shooter*

3. During the 2017–18 school year, has your school drilled students on the use of the following emergency procedures?

• Please respond to each of these according to the definitions provided. You may click on the asterisked words for their definitions, or click <definitions>.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: evacuation*, lockdown*, shelter-in-place*

4. During the 2017–18 school year, did your school have any activities that included the following components for students?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: bullying*, violence*, restorative circle*

5. During the 2017–18 school year, did your school have a threat assessment team* or any other formal group of persons to identify students who might be a potential risk for violent or harmful behavior (toward themselves or others)?

• If the respondent selects “No” the survey should skip to item 7. NOTE any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

ERROR will occur if the respondent tries to continue to the next question without providing an answer. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: threat assessment team*

6. During the 2017–18 school year, how often did your school’s threat assessment team* formally meet?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: threat assessment team*

7. During the 2017–18 school year, did your school have any recognized student groups with the following purposes?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: sexual orientation*, sexual identity*

Task 3. Moving forward without answering a question

Let’s stop here for a moment. Let’s now pretend that you didn’t want to answer the next question.

If you didn’t want to answer the next question, what would you do?

Is it clear that you can move forward in the survey without answering a question?

[If the user is still on the item page:] Now, please answer this question and then continue on with the survey.

[If the user has navigated away from the item page: While ideally we would like the user to answer these questions, if the user has moved forward as they click around, it is not necessary to have them go back and answer a missed question.]

SECTION: Parent and Community Involvement at School

8. Which of the following does your school do to involve or help parents?

9. What is your best estimate of the percentage of students who had at least one parent or guardian participating in the following events during the 2017–18 school year?

10. During the 2017–18 school year, were any of the following community and outside groups involved in your school’s efforts to promote safe, disciplined, and drug-free schools?

SECTION: School Security Staff

11. During the 2017–18 school year, did you have any sworn law enforcement officers (including School Resource Officers*) present at your school* at least once a week?

• Do not include security guards or other security personnel who are not sworn law enforcement in your response to this item; information on additional security staff is gathered in a later item.

• If the respondent selects “No” the survey should skip to item 19. NOTE any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

ERROR will occur if the respondent tries to continue to the next question without providing an answer. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*, at your school*

12. Were sworn law enforcement officers (including School Resource Officers*) used at least once a week in or around your school at the following times?

• Do not include security guards or other security personnel who are not sworn law enforcement in your response to this item; information on additional security staff is gathered in a later item.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*

13. Did any of the sworn law enforcement officers (including School Resource Officers*) at

your school* routinely:

• Do not include security guards or other security personnel who are not sworn law enforcement in your response to this item; information on additional security staff is gathered in a later item.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*, at your school*, firearm*

14. Did these sworn law enforcement officers (including School Resource Officers*) participate in the following activities at your school*?

• Do not include security guards or other security personnel who are not sworn law enforcement in your response to this item; information on additional security staff is gathered in a later item.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*, at your school*

15. During the 2017–18 school year, did your school have a sworn law enforcement officer (including School Resource Officers*) present for all instructional hours every day that school was in session?

• Include officers who are used as temporary coverage while regularly assigned officers are performing duties external to the school (such as attending court) or during these officers’ personal leave time.

• Check "No" if your school does not have officer coverage while regularly assigned officers are performing duties external to the school (such as attending court) or during these officers’ personal leave time.

• Do not include security guards or other security personnel who are not sworn law enforcement in your response to this item; information on additional security staff is gathered in a later item.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*

16. During the 2017–18 school year, did your school or school district have any formalized policies

or written documents (e.g., Memorandum of Use, Memorandum of Agreement) that outlined the

roles, responsibilities, and expectations of sworn law enforcement officers (including School

Resource Officers*) at school?

• If the respondent selects “No” the survey should skip to item 18. NOTE any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

ERROR will occur if the respondent tries to continue to the next question without providing an answer. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*

17. Did these formalized policies or written documents include language defining the role of sworn law enforcement officers (including School Resource Officers*) at school in the following areas?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*, firearm*, arrest*

18. How many of the following were present in your school at least once a week?

• If an officer works full-time across various schools in the district, please count this officer as “part-time” for your school.

• Do not include security guards or other security personnel who are not sworn law enforcement in your response to this item; information on additional security staff is gathered in a later item.

• If none, please select the None box.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box or do they enter “0” into the write-in box?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*

19. Aside from sworn law enforcement officers (including School Resource Officers*), how many additional security guards or security personnel were present in your school at least once a week?

• If a security guard or other security personnel works full-time across various schools in the district, please count this person as “part-time” for your school.

• If none, please select the None box.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: School Resource Officer*

Task 4. Log out of survey and log back in

Let’s stop here for a moment.

Now, pretend that you didn’t have time to finish the survey right now and wanted to finish it later. What would you do?

Record what page was on the screen when the user logged out: ________________________

NOTE what respondent does next.

[If the participant just stops but does not try to exit the survey.]

Please describe what you would do if you had to leave your desk to go to a meeting?

Please log out of the survey.

[After respondent has logged out of the survey] Now, please log back into the survey.

NOTE what respondent does next. Does he or she try to find their PIN?

[If respondent did not write down or memorize PIN]

I see that you don’t have your PIN. Was it clear that you would not be able to return to the survey if you don’t have your PIN?

Do you think you will still be able to complete the survey?

NOTE how participant would respond if they did not have a PIN.

• If respondent indicates that they think they are unable to complete the survey: Why do you think that?

• If respondent indicates that they think they can still complete the survey: What would you do to complete the survey?

[If respondent did write down their PIN]

Let’s pretend that you don’t have your PIN. What would you do?

NOTE what respondent would do if he or she did not have his or her PIN. Are they able to re-set the PIN correctly and easily?

[Once the respondent has successfully logged back into the survey].

Record what page was on the screen when the user logged back in:_____________________

NOTE if the survey sends the respondent to a different question than the one they just finished. Does respondent have any difficulty getting back to where they left off?

Please continue answering the questions.

SECTION: School Mental Health Services

20. During the 2017–18 school year, did your school provide diagnostic mental health assessments* (e.g., psychological/psychiatric diagnostics assessments) to evaluate students for mental health disorders*?

• Include only assessments conducted by a licensed mental health professional*.

• Include services that were provided at school* as well as services provided through a contract the school has with an outside provider.

• If the respondent selects “No” the survey should skip to item 22. NOTE any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

ERROR will occur if the respondent tries to continue to the next question without providing an answer. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: diagnostic mental health assessment*,

mental health disorder*, mental health professional*, at school*

21. Were diagnostic mental health assessment* services provided to students from your school in the following locations?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: diagnostic mental health assessment*,

mental health professional*, at school*

22. During the 2017–18 school year, did your school provide treatment* (e.g., psychotherapy, medication) to students for mental health disorders*?

• Include only treatment* provided by a licensed mental health professional*.

• Include services that were provided at school* as well as services provided through a contract the school has with an outside provider.

• If the respondent selects “No” the survey should skip to item 24. NOTE any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

ERROR will occur if the respondent tries to continue to the next question without providing an answer. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: treatment*, mental health disorder*,

mental health professional*, at school*

23. Were treatment* services provided to students from your school in the following locations?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: treatment*, mental health professional*,

at school*

24. During the 2017–18 school year, to what extent did the following factors limit your school’s efforts to provide mental health services to students?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: mental health disorder*,

diagnostic mental health assessment*, treatment*, mental health disorder*

SECTION: Staff Training and Practices

25. During the 2017–18 school year, did your school or school district provide any of the following for classroom teachers or aides?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: violence*, cyberbullying*, bullying*,

mental health disorders*

26. To the best of your knowledge, during the 2017–18 school year, were there any staff at your school* who legally carried a firearm* on school property?

• Exclude sworn law enforcement officers (including School Resource Officers*) or other security guards or personnel who carry firearms.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: at your school*, School Resource Officer*, firearm*

SECTION: Limitations on Crime Prevention

27. To what extent do the following factors limit your school’s efforts to reduce or prevent crime?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: special education student*

SECTION: Frequency of Crime and Violence at School

28. During the 2017–18 school year, have any of your school’s students, faculty, or staff died as a result of a homicide committed at your school*?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: at your school*

29. During the 2017–18 school year, has there been at least one incident at your school* that involved a shooting (regardless of whether anyone was hurt)? Please include those incidents that occurred at school*, regardless of whether a student or non-student used the firearm*.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: at your school*/at school*, firearm*

This next section of the questionnaire is going to ask for counts of specific incidents. Please estimate your answer when responding to these items.

SECTION: Incidents

30. Please record the number of incidents that occurred at school* during the 2017–18 school year for the offenses listed below. (NOTE: For each of the sub-items, the number of total incidents recorded should be greater than or equal to the number of incidents reported for the number reported to police or other law enforcement.)

• If none, please select the None box.

Please provide information on:

• The number of incidents, not the number of victims or offenders.

• Recorded incidents, regardless of whether any disciplinary action was taken.

• Recorded incidents, regardless of whether students or non-students were involved.

• Incidents occurring before, during, or after normal school hours.

• For each sub-item, if the respondent enters a number greater than 0, a second row will appear. OBSERVE how respondent handles this situation.

• For each sub-item, an error should pop up if the respondent enters a number reported to police that is greater than the number of incidents they entered. OBSERVE how respondent handles error.

• Are there any instances in which a respondent receives questions that are not applicable to him or her? When this happens, can he or she find his or her way out?

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: at school*, rape*, sexual assault*, robbery*, weapon*, physical attack or fight*, theft/larceny*, firearm/explosive device*, vandalism*

31. During the 2017–18 school year, how many hate crimes* occurred at your school*?

• If none, please select the None box.

• If the respondent selects “None” or enters “0”, the survey should skip to item 33. NOTE any difficulty with skips. Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

ERROR will occur if the respondent tries to continue to the next question without providing an answer. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: hate crime*, at your school*

32. To the best of your knowledge, were any of these hate crimes* motivated by the offender’s bias against the following characteristics or perceived characteristics?

• If a hate crime* was motivated by multiple characteristics, answer “Yes” for each that applies.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: hate crimes*, sexual orientation*,

gender identity*

33. To the best of your knowledge, during the 2017–18 school year, have there been any incidents of sexual misconduct* between a staff member and a student at your school?

• Report on misconduct between staff and students whether or not the incidents occurred at school or away from school.

• Sexual assault* and rape* are both forms of sexual misconduct, therefore, some incidents of staff-student behavior may be reported in response to previous questions.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: sexual misconduct*, sexual assault*, rape*

34. Please record the number of arrests* that occurred at your school during the 2017–18 school year. Please include all arrests* that occurred at school*, regardless of whether a student or non-student was arrested.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: arrest*, at school*

Let’s stop here for a moment.

In your regular setting, where would you obtain the counts provided in the previous section?

[If the respondent noted they would consult records, such as a database or other file on their computer, ask:]

Without opening the ________, can you walk me through what you would have to do in order to access the files with the information?

NOTE/OUTLINE respondent’s process.

[Once respondent has completed the process]

Would it be easy or difficult to navigate between the questionnaire and the _________? [If respondent says “difficult”] Can you tell me a little more about why you said difficult?

Please continue answering the questions.

SECTION: Disciplinary Problems and Actions

35. To the best of your knowledge, how often do the following types of problems occur at your school*?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: at your school*, bullying*, sexual harassment*, harassment*, sexual orientation*, gender identity*, gang*

36. To the best of your knowledge, thinking about problems that can occur anywhere (both at your school* and away from school), how often do the following occur?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: at your school*, cyberbullying*

37. During the 2017–18 school year, did your school allow for the use of the following disciplinary actions? If "yes," were the actions used this school year?

• If “yes” is answered for any sub-item in the first column (“Does your school allow for the following?”), then a second column will appear (“If "Yes," was the action used this school year?”). OBSERVE how respondent handles this situation.

• Are there any instances in which a user receives questions that are not applicable to them? When this happens, can he or she find his or her way out?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: specialized school*

38. During the 2017–18 school year, how many students were involved in committing the following offenses, and how many of the following disciplinary actions were taken in response?

• If none, please select the None box.

Please follow these guidelines when determining the number of offenses and disciplinary actions:

• If more than one student was involved in an incident, please count each student separately when providing the number of disciplinary actions.

• If a student was disciplined more than once, please count each offense separately (e.g., a student who was suspended five times would be counted as five suspensions).

• If a student was disciplined in two different ways for a single infraction (e.g., the student was both suspended and referred to counseling), count only the most severe disciplinary action that was taken.

• If a student was disciplined in one way for multiple infractions, record the disciplinary action for only the most serious offense.

• OBSERVE how respondent reacts to the format of this item.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

• ERROR will occur if the “total” row is marked “None” but any of the subsequent rows are greater than zero. OBSERVE how respondent handles error(s). OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: firearm/explosive device*, weapon*,

physical attack or fight*, specialized school*

39. During the 2017–18 school year, how many of the following occurred?

• If none, please select the None box.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

• ERROR will occur if the answer provided in item 39a is less than the sum of entries in column 2 of item 38 or if the answer provided in item 39b is less than the sum of entries in column 3 of item 38. OBSERVE how respondent handles error(s). OBSERVE how respondent closes out of the error pop-up box.

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: specialized school*

TASK 5. FAQ section

Now let’s stop here for a moment. I’d like to ask you some questions about what you would do if you encountered a certain situation while completing the survey.

If you had a question about the survey, what would you do?

OBSERVE whether the respondent has any difficulties finding the FAQ section. NOTE if the respondent ever used (or noticed) the FAQ section while completing the survey thus far.

[If the respondent did not see the FAQ link, help them find it.]

Can you find a place on this website where you could go to find an answer if you had a question while completing the survey?

NOTE any reaction to the FAQ section.

If you couldn’t find the answer to your question in the FAQ section, what would you do?

Can you find someone to contact if you had a question?

OBSERVE whether respondent can find Census contact information.

Now let’s finish up the survey. There are only a few questions left.

SECTION: School Characteristics: 2017–18 School Year

40. As of October 1, 2017, what was your school’s total enrollment?

• ERROR will occur if the respondent tries to continue to the next question without providing an answer or if he or she tries to enter “0”. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

41. What percentage of your current students fit the following criteria?

• If none, please select the None box.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

If a respondent looks up the definition for a term, CIRCLE the term below.

For this question, definitions are available for: special education student*

42. What is your best estimate of the percentage of your current students who meet the following criteria?

• If none, please select the None box.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

43. How many classroom changes do most students make in a typical day?

• Count going to lunch and then returning to the same or a different classroom as two classroom changes. Do not count morning arrival or afternoon departure.

• If none, please select the None box.

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

ERROR will occur if the respondent tries to enter a number greater than 20. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

44. How would you describe the crime level in the area(s) in which your students live?

45. How would you describe the crime level in the area where your school is located?

46. Which of the following best describes your school?

• If respondent selects the “Other” option, OBSERVE how the respondent handles the “Please specify” portion. Or if respondent does not select the “Other” option, please note any reaction to the greyed-out box.

47. What is your school’s average daily attendance?

• OBSERVE how respondent handles a situation in which he or she wants to respond with “zero.” Does the respondent click the “None” box as instructed or do they enter “0” into the write-in box?

48. During the 2017–18 school year, how many students transferred to or from your school after the start of the school year? Please report on the total mobility, not just transfers due to disciplinary actions.

• If a student transferred more than once in the school year, count each transfer separately.

• If none, please select the None box.

• ERROR will occur if the number reported by the respondent does not logically align with their response to item 39b. OBSERVE how respondent handles error. OBSERVE how respondent closes out of the error pop-up box.

SECTION: Contact Information

Please provide the following dates:

• OBSERVE how respondent reacts to the format of this item. Does he or she enter the month and date correctly?

Which of the following grades are offered in this school?

• Select all that apply.

• OBSERVE whether respondent understands that he or she can mark multiple grades.

Name of person completing this questionnaire

• OBSERVE how respondent reacts to the format of this item. Are there enough character spaces?

Telephone number

• OBSERVE how respondent reacts to the format of this item.

Title/position

• If respondent selects the “Other” option, OBSERVE how the respondent handles the “Please specify” portion. Or if respondent does not select the “Other” option, please note any reaction to the greyed-out box.

Number of years at this school

• OBSERVE how respondent reacts to the format of this item.

Best days and times to reach you (in case we have further questions)

• OBSERVE how respondent reacts to the format of this item. Are there enough character spaces?

E-mail address

• OBSERVE how respondent reacts to the format of this item. Are there enough character spaces?

How long did it take you to complete this form, not counting interruptions?

• Please record the time in minutes (e.g., 55 minutes, 65 minutes).

Submit page

• OBSERVE whether the respondent clicks the “Submit Data” button.

Thank-you page

• OBSERVE whether the respondent recognizes and/or clicks the “print survey” PDF icon.

STEP 4: DEBRIEF

Thank you for completing the SSOCS questionnaire. Now we would like you to fill out a short survey about your experiences with the web survey you just completed. There are no right or wrong answers. Your responses are important to us and will inform revisions to the web survey you just used. We may ask you some follow-up questions about your responses so that we can get your feedback on how best to improve the web survey.

[Give respondent the QUIS survey.]

Review QUIS responses – probe any items rated less than 5

Could you tell me more about why you rated ­­­­­­­________________ as a _______?

Overall, what was your impression of the survey?

What features did you find the most helpful to you?

What features did you find the least helpful to you?

Are there any other features you would like to see in the platform?

Is there anything else you would like to mention about the survey that we haven’t had a chance to talk about?

STEP 5: WRAP & THANK YOU

Okay, we are all finished. Thank you for your participation today. The information you’ve provided will be very helpful for making this web survey better.

[Please give the respondent incentive envelope or let them know that one will be sent within 2 business days via mail, and thank them for their time and effort.

If in person, collect materials and escort respondent to the lobby or street.]

Appendix L– Questionnaire for User Interaction Satisfaction (QUIS)

Please read each statement carefully. Rate each statement using the words listed above the endpoints of the scale that most appropriately reflect your impressions about responding to this web survey. (Not applicable = NA)

Overall reactions to the interface	terrible							wonderful
	1	2	3	4	5	6	7	8	9	NA
	frustrating							satisfying
	1	2	3	4	5	6	7	8	9	NA
	difficult							easy
	1	2	3	4	5	6	7	8	9	NA
Getting started	difficult							easy
	1	2	3	4	5	6	7	8	9	NA
Arrangement of information on screen	illogical							logical
	1	2	3	4	5	6	7	8	9	NA
Screen layouts were helpful	never							always
	1	2	3	4	5	6	7	8	9	NA
Characters on the computer screen	hard to read							easy to read
	1	2	3	4	5	6	7	8	9	NA
Use of bolding	helpful							unhelpful
	1	2	3	4	5	6	7	8	9	NA
Correcting your mistakes	difficult							easy
	1	2	3	4	5	6	7	8	9	NA
Error messages	helpful							unhelpful
	1	2	3	4	5	6	7	8	9	NA
Phrasing of error messages	unpleasant							pleasant
	1	2	3	4	5	6	7	8	9	NA
Ability to undo operations	inadequate							adequate
	1	2	3	4	5	6	7	8	9	NA

39