National Center for Education Statistics
Volume I
Supporting Statement
2019 School Crime Supplement to
the National Crime Victimization Survey (SCS:19/ NCVS)
Spanish Cognitive Interviews
OMB# 1850-0803 v.223
Attachments:
Attachment I – Recruitment Advertisements
Attachment II – Recruitment Protocol and Eligibility Screener
Attachment III – Sample Consent Forms
February 2018
The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting testing such as pilot tests, focus groups, and cognitive interviews.
This request is to conduct Spanish language recruitment and cognitive interviews designed to evaluate new and revised questionnaire items for the 2019 School Crime Supplement to the National Crime Victimization Survey (SCS:19/NCVS). This package presents the Spanish translation of question wording to be tested and describes plans and procedures for conducting the cognitive interviews. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, U.S.C. 20 §9543).
The School Crime Supplement (SCS) to the National Crime Victimization Survey (NCVS) was co-designed by the National Center for Education Statistics (NCES) and the Bureau of Justice Statistics (BJS). The SCS collects data on school-related topics, including alcohol and drug availability, fighting, bullying and hate related behaviors, and fear and avoidance behaviors from students age 12 to 18 in U.S. public and private elementary, middle, and high schools. To date, the SCS was conducted in 1989, 1995, 1999, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015, and 2017. The questionnaire has been available in Spanish since 2011.
In 2013, NCES and BJS initiated revisions of the SCS, specifically adjustments to question wording to be consistent with the Centers for Disease Control and Prevention’s (CDC) definition of “bullying,” including “cyber-bullying,” as well as changes to wording based on other research. The CDC modified its definition of bullying to include two additional components: repetition and a power differential. A technical review panel was convened to recommend changes to the SCS that could meet the CDC’s definitional requirements. The 2015 SCS questionnaire included two versions of the question used to measure the prevalence of bullying. One version (V1) maintained the original question on bullying (consistent with 2013 SCS question to help preserve the time trend) and added two new follow-up questions to capture the two new components. In the other version (V2), respondents were asked a single new question on bullying that included the two new components. Analyses of the two versions of bullying questions used in the 2015 SCS produced differing estimates of bullying. Additionally, the estimate produced by the new item (V2) was significantly lower than the 2013 and the 2015 estimates using the follow-up items to the original wording (V1). This suggested that the items were not performing well, and should be revised to improve performance.
In 2016, the questions were revised and we conducted cognitive interviews (OMB# 1850-0803 v.154 and 161) in English to test the revised bullying questions prior to fielding the 2017 SCS. Preliminary analysis of the results indicate that the 2017 version of the bullying questions performed well, producing estimates of bullying closer to those produced in prior years. Research has shown that including a term like “bullying,” which has a variety of colloquial meanings, in the question wording has the potential to influence measurement error. The next step in improving the questions is to remove the terms “bullying” and “bullied,” and use a set of behavior-specific questions to measure the different components included in the bullying definition.
In late 2017 and early 2018, the first two rounds of cognitive interviews (OMB# 1850-0803 v.213) were conducted in English to test questions with the terms “bullying” and “bullied” removed. This request is to conduct the final 3 rounds of cognitive interviews with Spanish-dominant middle school and high school students to evaluate the current translation of items, and the translation of new items for the SCS on the topic of bullying in schools. While the testing focuses on questions about bullying, probing questions will also be asked about other 2019 SCS items that may benefit from revision, particularly those which have not been previously tested. The cognitive interviews will enable the team to identify problems with question wording, translation, and suggest revisions to problematic questions.
Exhibit 1 summarizes the number of interviews to be conducted within each round, as well as the recruitment burden. We will only test one version of the survey in each round. The cognitive testing will use an iterative process. Data from the interviews will be analyzed after each round to identify problematic questions and the results will be considered together with the results of the ongoing cognitive interview testing of these items in English to address recommended changes from testing in both languages. The questions tested in the next round will be revised and tested in a subsequent round.
To test the 2019 SCS instrument, we will recruit a total of 18 Spanish-speaking students ages 12-18 who currently attend public or private middle schools and high schools. We will strive to recruit a sample with diverse demographic characteristics and school levels (middle school and high school). In addition to covering a spectrum of demographic and school level characteristics, we aim to recruit a sample of students who have experienced negative interactions with other students from their school that may count as bullying as well as students who have not. Spanish language testing will run concurrently with Rounds 3, 4, and 5 of the already approved testing in English (OMB# 1850-0803 v.213).
Each interview will be 60 minutes in length and will allow time to complete the SCS survey and administer a set of in-depth retrospective probes about the questions in the bullying section. From prior experience, we anticipate needing to contact approximately 72 individuals to yield the 18 desired interviews to account for ineligible respondents and cancellations. We anticipate needing a higher number of contacts per recruited student because this is a hard-to-survey population that requires specific characteristics of recruited students, such as being a Spanish-dominant speaker, being between the ages of 12 and 18, and feeling comfortable to talk about negative interactions with other students at school.
Exhibit 1. Estimated Respondent Burden by Round of Cognitive Interviews
Response Type |
Respondents per Round |
Total # of respondents |
Total # of responses |
Time Estimate (minutes) per respondent |
Total burden hours |
||
Round 1 |
Round 2 |
Round 3 |
|||||
Student recruitment |
24 |
24 |
24 |
72 |
72 |
10 |
12 |
SCS Questionnaire |
|||||||
Middle School Students |
3 |
3 |
3 |
9 |
9 |
60 |
9 |
High School Students |
3 |
3 |
3 |
9 |
9 |
60 |
9 |
Total |
24 |
24 |
24 |
72 |
90 |
-- |
30 |
Cognitive interview participants will be recruited using partnerships with student organizations in Maryland, Virginia, and the District of Columbia; through listservs; and through personal networks. See Attachments I and II for the English and Spanish versions of the language to be used in recruitment advertisements and for the eligibility screener. The respondents will receive materials by email or on paper if their interview is scheduled in person, to remind them of their interview time and any other directions or information they need.
Bilingual researchers from the Center for Survey Measurement (CSM) at the U.S. Census Bureau will carry out the cognitive interviews in Spanish. Between two and three staff members will be trained to conduct the interviews to allow flexibility in interview scheduling. All interviewers will be required to demonstrate a strong understanding of the interview protocol before beginning interviews. With respondent permission, the interviews will be audio-recorded so that the responses may be reviewed as needed during analysis. The CSM project leader will conduct interviewer training, supervise staff, and monitor data quality throughout the data collection period.
During each round of testing, interviewers will administer the survey through item 32 in Section G, on fighting, bullying, and hate behaviors (see the cognitive interview protocol in Attachment IV). The interviewer will observe the respondent as they answer the survey items, noting any questions or problems the respondent has with regards to particular items. After completing these sections, respondents will be asked a set of probing questions about some of the survey items, and about any questions or problems the respondent had when answering the main survey questions. Interviewers will then administer the remaining survey items. Again, interviewers will ask respondents a set of probing questions about some of the survey items in Q33 – Q46b and the translations of terms to Spanish, followed by a set of debriefing questions. These items will be iteratively tested within each of the three rounds, with the protocol and question wording being adapted based on the results from the previous round. In addition, the Spanish language team will work closely with the English team to adapt each round of testing to the results of the both Spanish and English testing.
Cognitive interviewing techniques allow researchers to evaluate potential response error and to assure that the survey provides valid data. In general, the goal of cognitive testing is to assess the respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases in the item. Data from cognitive interviews can identify potentially problematic questions that are not understood as intended. This testing can also examine the respondents’ retrieval of relevant information from memory, decision processes involved with answering a question, and questions that are difficult to answer due to cognitive burden. The cognitive interviews will assess issues such as:
The subject’s understanding of terms in the survey
How confident the subject is in their response
How they remembered the information they provided in factual questions
Whether they found a response choice that fit their answer
How easy or difficult it is to answer a question
Issues with sensitive questions
Consistency of answers within the questionnaire and in comparison to the expected range of answers
Cognitive interview participants will be informed that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Participants will also be advised that the interview will be recorded and that the audio recording will only be reviewed for the purposes of this study. Lastly, participants will be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a respondent says.
Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials together. The respondent ID will not be linked to the respondent in any way or form. If respondents are under the age of 18, their parents will be provided with a parental consent form; respondents who are 18 will be provided with the standard consent form used for adults (see Attachment III). The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study.
To encourage participation and thank them for their time and effort, $25 will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, they will also receive $25 cash as a thank you for their time and effort and for allowing their child to participate in the study.
The cost of conducting the cognitive interviews will be $65,000, under the NCES contract to CSM at the U.S. Census Bureau.
Recruit participants through networks and advertisements |
February 2018– May 18, 2018 |
Conduct Spanish Round 1 cognitive testing (concurrent with English testing) |
February 26 – March 16, 2018 |
Iterative revisions to item wording |
March 19 – April 6, 2018 |
Conduct Spanish Round 2 cognitive testing (concurrent with English testing) |
April 9 – April 20, 2018 |
Iterative revisions to item wording |
April 23 – May 4, 2018 |
Conduct Spanish Round 3 cognitive testing (concurrent with English testing) |
May 7 – May 18, 2018 |
Analysis and Final Recommendations |
May 21 – June 1, 2018 |
Final Wording for Questionnaire |
June 1, 2018 |
Final Report |
July 30, 2018 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Herschel Sanders |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |