Justification

National Center for Education Statistics

Volume I

Supporting Statement

2019 School Crime Supplement to

the National Crime Victimization Survey (SCS:19/ NCVS)

Cognitive Interviews

OMB# 1850-0803 v.213

Attachments:

Attachment I – Recruitment Advertisements

Attachment II – Recruitment Protocol and Eligibility Screener

Attachment III – Sample Consent Forms

Attachment IV – Cognitive Interview Protocol and Questionnaire Items

Attachment V – School Crime Supplement, Parent Brochure

Attachment VI – Parent Brochure Protocol

October 2017

Submittal-Related Information

The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting testing such as pilot tests, focus groups, and cognitive interviews.

This request is to conduct recruitment and cognitive interviews designed to evaluate new and revised questionnaire items for the 2019 School Crime Supplement to the National Crime Victimization Survey (SCS:19/NCVS). This package presents the question wording to be tested and describes plans and procedures for conducting the cognitive interviews. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, U.S.C. 20 §9543).

Background

The School Crime Supplement (SCS) to the National Crime Victimization Survey (NCVS) was co-designed by the National Center for Education Statistics (NCES) and the Bureau of Justice Statistics (BJS). The SCS collects data on school-related topics, including alcohol and drug availability, fighting, bullying and hate related behaviors, and fear and avoidance behaviors from students age 12 to 18 in U.S. public and private elementary, middle, and high schools. To date, the SCS was conducted in 1989, 1995, 1999, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015, and 2017.

In 2013, NCES and BJS initiated revisions of the SCS, specifically adjustments to question wording to be consistent with the Centers for Disease Control and Prevention’s (CDC) definition of “bullying,” including “cyber-bullying,” as well as changes to wording based on other research. The CDC modified its definition of bullying to include two additional components: repetition and a power differential. A technical review panel was convened to recommend changes to the SCS that could meet the CDC’s definitional requirements. The 2015 SCS questionnaire included two versions of the question used to measure the prevalence of bullying. One version (V1) maintained the original question on bullying (consistent with 2013 SCS question to help preserve the time trend) and added two new follow-up questions to capture the two new components. In the other version (V2), respondents were asked a single new question on bullying that included the two new components. Analyses of the two versions of bullying questions used in the 2015 SCS produced differing estimates of bullying. Additionally, the estimate produced by the new item (V2) was significantly lower than the 2013 and the 2015 estimates using the follow-up items to the original wording (V1). This suggested that the items were not performing well, and should be revised to improve performance.

In 2016, the questions were revised and we conducted cognitive interviews (OMB# 1850-0803 v.154 and 161) to test the revised bullying questions prior to fielding the 2017 SCS. Preliminary analysis of the results indicate that the 2017 version of the bullying questions performed well, producing estimates of bullying closer to those produced in prior years. Research has shown that including a term like “bullying,” which has a variety of colloquial meanings, in the question wording has the potential to influence measurement error. The next step in improving the questions is to remove the terms “bullying” and “bullied,” and use a set of behavior-specific questions to measure the different components included in the bullying definition.

This request is to conduct 5 rounds of cognitive interviews with middle school and high school students to evaluate the new items for the SCS on the topic of bullying in schools. While the testing focuses on questions about bullying, probing questions will also be asked about other 2019 SCS items that may benefit from revision. The cognitive interviews will enable the team to identify problems with question wording and suggest revisions to problematic questions. Additionally, parents and guardians who bring their participating child to the cognitive interview will be invited to provide feedback on the 2019 SCS parent/guardian brochure designed to answer questions and encourage survey participation. The interviews and feedback will be used to develop more effective study materials.

Study Design, Context, and Respondent Burden

Exhibit 1 summarizes the number of interviews to be conducted within each round, as well as the recruitment burden. We will only test one version of the survey in each round. The cognitive testing will use an iterative process. Data from the interviews will be analyzed after each round to identify problematic questions. Those questions will be revised and tested in the subsequent round.

To test the 2019 SCS instrument, we will recruit a total of 50 students ages 12-18 who currently attend public or private middle schools and high schools. We will strive to recruit a sample with diverse demographic characteristics and school levels (middle school and high school). In addition to covering a spectrum of demographic and school level characteristics, we aim to recruit a sample of students who have experienced negative interactions with other students from their school that may count as bullying as well as students who have not. Testing of the survey will be conducted in Spanish concurrently with rounds 3 and 4. Details about the Spanish testing will be submitted under a new cognitive interviews request (OMB# 1850-0803 v.TBD) at a later date.

Each interview will be 60 minutes in length and will allow time to complete the SCS survey and administer a set of in-depth retrospective probes about the questions in the bullying section. From prior experience, we anticipate needing to contact approximately 135 individuals to yield the 50 desired interviews to account for ineligible respondents and cancellations.

A majority, if not all, students who participate in the interviews will be accompanied by a parent or guardian. We will ask those parents and guardians if they would like to participate in a short interview while they wait for the student. The parent/guardian interview will be 15 minutes in length. Respondents will be shown a copy of the parent brochure that is given to parents/guardians of students selected for the SCS sample. They will then be asked a series of probing questions designed to elicit feedback on the content and design of the brochure, as well as their overall reactions to the brochure. See attachments V and VI for the parent brochure and the protocol for the parent/guardian interview. No additional recruiting efforts are necessary for these interviews. Every parent or guardian accompanying a student will be asked if they would like to participate, yielding up to 50 interviews.

Exhibit 1. Estimated Respondent Burden by Round of Cognitive Interviews

Response Type	Respondents per Round					Total # of respondents	Total # of responses	Time Estimate (minutes) per respondent	Total burden hours
	Round 1	Round 2	Round 3	Round 4	Round 5
Student recruitment	27	27	27	27	27	135	135	10	23
SCS Questionnaire
Middle School Students	5	5	5	5	5	25	25	60	25
High School Students	5	5	5	5	5	25	25	60	25
Parents and Guardians	10	10	10	10	10	50	50	15	13
Total	37	37	37	37	37	185	235	--	86

Cognitive interview participants will be recruited using partnerships with student organizations in Maryland, Virginia, and the District of Columbia; through listservs; and through personal networks. See Attachments I and II for the language to be used in recruitment advertisements and for the eligibility screener. The respondents will receive materials by email or on paper if their interview is scheduled in person, to remind them of their interview time and any other directions or information they need.

Researchers from the Center for Survey Measurement (CSM) at the U.S. Census Bureau will carry out the cognitive interviews. Between four and seven staff members will be trained to conduct the interviews to allow flexibility in interview scheduling. All interviewers will be required to demonstrate a strong understanding of the interview protocol before beginning interviews. With respondent permission, the interviews will be audio-recorded so that the responses may be reviewed as needed during analysis. The CSM project leader will conduct interviewer training, supervise staff, and monitor data quality throughout the data collection period.

During each round of testing, interviewers will administer the survey through item 32 in Section G, on fighting, bullying, and hate behaviors (see the cognitive interview protocol in Attachment IV). The interviewer will observe the respondent as they answer the survey items, noting any questions or problems the respondent has with regards to particular items. After completing these sections, respondents will be asked a set of probing questions about some of the survey items, and about any questions or problems the respondent had when answering the main survey questions. Interviewers will then administer the remaining survey items. Again, interviewers will ask respondents a set of probing questions about some of the survey items in Q33 – Q46b, followed by a set of debriefing questions. These items will be iteratively tested within each of the five rounds, with the protocol and question wording being adapted based on the results from the previous round.

Cognitive interviewing techniques allow researchers to evaluate potential response error and to assure that the survey provides valid data. In general, the goal of cognitive testing is to assess the respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases in the item. Data from cognitive interviews can identify potentially problematic questions that are not understood as intended. This testing can also examine the respondents’ retrieval of relevant information from memory, decision processes involved with answering a question, and questions that are difficult to answer due to cognitive burden. The cognitive interviews will assess issues such as:

• The subject’s understanding of terms in the survey

• How confident the subject is in their response

• How they remembered the information they provided in factual questions

• Whether they found a response choice that fit their answer

• How easy or difficult it is to answer a question

• Issues with sensitive questions

• Consistency of answers within the questionnaire and in comparison to the expected range of answers

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Participants will also be advised that the interview will be recorded and that the audio recording will only be reviewed for the purposes of this study. Lastly, participants will be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a respondent says.

Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials together. The respondent ID will not be linked to the respondent in any way or form. If respondents are under the age of 18, their parents will be provided with a parental consent form; respondents who are 18 will be provided with the standard consent form used for adults (see Attachment III). The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study.

Estimate of Costs for Recruiting and Paying Respondents

To encourage participation and thank them for their time and effort, $25 will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, they will also receive $25 cash as a thank you for their time and effort and for allowing their child to participate in the study.

Cost to the Federal Government

The cost of conducting the cognitive interviews will be $130,000, under the NCES contract to CSM at the U.S. Census Bureau.

Project Schedule

Recruit participants through networks and advertisements	November 2017 – May 16, 2018
Conduct Round 1 cognitive testing	December 4 – December 22, 2017
Iterative revisions to item wording	December 26, 2017 – January 12, 2018
Conduct Round 2 cognitive testing	January 15 – February 2, 2018
Iterative revisions to item wording	February 5 – February 23, 2018
Conduct Round 3 cognitive testing	February 26 – March 16, 2018
Iterative revisions to item wording	March 19 – April 6, 2018
Conduct Round 4 cognitive testing	April 9 – April 20, 2018
Iterative revisions to item wording	April 23 – May 4, 2018
Conduct Round 5 cognitive testing	May 7 – May 18, 2018
Analysis and Final Recommendations	May 21 – June 1, 2018
Final Wording for Questionnaire	June 1, 2018
Final Report	July 30, 2018