Justification

Background, Study Rationale, and Results of Round 1

The ED School Climate Surveys (EDSCLS) are a suite of survey instruments being developed for schools, school districts, and states by the U.S. Department of Education’s National Center for Education Statistics (NCES). This national effort extends current activities that measure school climate, including the state-level efforts of Safe and Supportive Schools (S3) grantees, which were awarded funds in 2010 by the Department of Education’s Office of Safe and Healthy Students (OSHS) to improve school climate. Through the EDSCLS, schools nationwide will have access to survey instruments and a survey platform that will allow for the collection and reporting of school climate data across stakeholders at the local level. The surveys can be used to produce school-, district- and state-level scores on various indicators of school climate from the perspectives of students, teachers and staff, principals, and parents and guardians. NCES will also provide benchmark data, collected from a nationally representative sample of schools across the United States, to facilitate comparisons between school climate scores at the local and national levels (OMB# 1850-0923 v.1).

As part of EDSCLS development process, NCES conducted cognitive interviews in 2014 (OMB# 1850-0803 v.102 and 107) and a pilot test in 2015 (OMB# 1850-0803 v.119, 126, and 129). In 2015, the Office of Safe and Healthy Students (OSHS) requested that new items addressing school climate and sexual assault be added to the EDSCLS questionnaires. School communities nationwide have identified sexual assault as an issue that students and schools are increasingly facing. OSHS has requested for EDSCLS to produce data to assist schools in creating supports and protective factors for reducing sexual assault as part of creating a safe and healthy learning environment. To test proposed new items, NCES conducted a targeted set of cognitive interviews in 2015 (OMB# 1850-0803 v.145).

Since the 2015 round of cognitive testing, OSHS and NCES have decided to continue refining the set of school climate and sexual assault items through additional development and cognitive testing. These items will not be added directly to the EDSCLS platform, which would make their administration mandatory, but instead they will be made available for individual schools and districts to download from the OSHS website for inclusion at their own discretion. Round 1 of testing was recently completed. This request is to cognitively test, during late fall and early winter of the 2016-17 school year, the second draft of the new items with two primary target groups: (1) students in grades 5-8 and 9-12, and (2) school teachers, other non-teaching staff, and administrators.

Results of the first round of the second set of testing are provided in Appendix S. The first round of testing with instructional and other staff demonstrated that the drafted questions worked well, but two fairly substantive issues were identified. First, for the prevalence questions, the use of the term “often” was confusing. Respondents required quantification of the term and struggled with how to answer these questions. They knew the behaviors occurred, but not the exact frequency. Second, some respondents felt that some of the questions were redundant and that it was difficult to discern what had changed in the questions. In response to these findings, modifications were made to the set of questions. Prevalence questions were revised to ask about the extent to which respondents feel that a particular behavior “is a problem at this school” rather than how often students experience it. In addition, the sexual harassment questions were streamlined into two questions asking about the set of behaviors at the school (originally split into 3 questions) and another asking about the set of behaviors outside of school (the original question did not differentiate between settings). Finally, additional question structure changes further differentiated settings. Other wording changes were recommended by the cognitive testing results – e.g., changed “sexual body parts” to “private body parts.”

The first round of testing with students found that the first draft of questions worked well with minor changes recommended. Similarly to school staff, students found it difficult to discern what had changed between questions. Asking whether or not a behavior is a problem may be too cognitively difficult for students, and using the qualifier “often” has now been changed to ask whether or not a behavior happens. In addition, the sexual harassment items have been streamlined to ask about behaviors at school and outside of school and a new question was added about sexual battery experienced outside of school because staff felt students would be a better source for this information. Similarly as in the questions for school staff, wording changes were made to the student questions based on the cognitive test results.

Type of Testing

We will conduct the second round of cognitive interviews using the revised set of items from those developed for the Bureau of Justice Statistics’ and the Office of Violence against Women’s National Campus Climate Survey Validation Study. Using a paper survey, the interviewer will use the same structured protocol as the first round, a one-on-one interview drawing on methods from cognitive science. The cognitive interviews will investigate the cognitive processes that respondents use to answer survey questions, and identify any problems of ambiguity or misunderstanding in question wording. The goal is to ensure that the final items are easily understood by respondents, with their interpretations consistently aligned with the concepts being measured.

We will use think-aloud interviewing and verbal probing techniques (also known as concurrent and retrospective recall probing, respectively). With think-aloud interviewing, respondents are explicitly instructed to think aloud (i.e., describe what they are thinking) as they work through items. With verbal probing techniques, the interviewer asks probing questions, as necessary, to clarify points that are not evident from the “think-aloud” process, or to explore additional issues that have been identified a priori as being of particular interest. Cognitive interview studies produce qualitative data in the form of verbalizations made by participants during the think-aloud phase and in response to the interviewer probes.

Survey Items

The student cognitive interviews will include 10 items: two items to provide context, and eight items that focus on sexual assault at the school. For other instructional and non-instructional staff, the cognitive interviews will cover 12 content-specific items. The items are provided in Volume II and recruiting materials are provided in Appendices A-R.

Sample and Recruitment Plans

Research Support Services (RSS), under subcontract to RTI International (RTI) will carry out the cognitive laboratory testing in November 2016 through February 2017. RSS will attempt to recruit, from the Chicago metropolitan area where the RSS office is located, participants representing a range of characteristics (including urban and suburban areas, students from a mix of grades, and staff serving in a variety of school roles), although the results will not explicitly measure differences by these characteristics. RSS will use multiple outreach methods and resources, such as marketing research companies, newspaper/internet ads, and contacts with community organizations (e.g., libraries and afterschool programs) to recruit participants. Paper flyers, ads, e-mails, and/or phone calls will be used to contact the potential participants.

Interested participants will be screened using a screener script to ensure that they meet the criteria for participation in the interviews. During this communication, the parent or guardian of any interested minors will be informed about the objectives, purpose, and participation requirements of the data collection effort, as well as the activities that it entails. Only after RSS has obtained written consent from the parent or guardian will the student be allowed to participate in the cognitive laboratory testing session. RSS will confirm the interview date, time, and location via e-mail, letter, or telephone. Principals, non-instructional staff, teachers, students, and the parents or legal guardians of participating students will complete consent forms at the time of the interview. Recruitment will continue until interviews are completed with 10 participants from each group. We anticipate recruiting about 32 potential participants per group to achieve the 10 completed interviews in each.

The cognitive laboratory testing will take place in a space rented by RSS or at a local library convenient to participants. Participants will first be welcomed, introduced to the interviewer and the observer (if an in-room observer is present), and told that they are there to help answer questions about how people answer school climate survey items. All participants will be reassured that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Interviewers will explain the think-aloud process and conduct a practice session with a sample question.

The think-aloud component of the cognitive interviews will use either (1) a concurrent think-aloud method in which the participant verbalizes his or her thoughts while working through a questionnaire or (2) a retrospective think-aloud method during which participants work through a questionnaire silently and then discuss their thoughts about the item content while working through it a second time. The cognitive interviews will also include a verbal probing component conducted after completion of the think-aloud portion for a given questionnaire. The verbal probes will include a combination of pre-planned item-specific questions, identified before the session as important and ad hoc questions that the interviewer identifies as important from observations during the interview, such as clarifications or expansions on points raised by the participant. To minimize the burden on the participant, efforts will be made to limit the number of verbal probes that can be used in any one session. The protocols will contain generic prompts to be applied flexibly by the interviewer to facilitate and encourage participants to verbalize their thoughts. For example: “I see you’re looking at the answer. What are you thinking?” or “Are there any questions or words that seem confusing here?” Observers will take notes on what participants say and the sessions will be audio recorded.

The results will again be compiled to identify patterns of responses and any item specific issues. This approach will help to ensure that the data are analyzed in a thorough and systematic way that enhances the identification of problems with items or tasks and provides recommendations for addressing them. Following this round of testing, the set of items will only be revised and retested if the second round results warrant. If required, up to 4 testing rounds have been planned. The future cognitive testing rounds will be submitted in a separate request to OMB under this 1850-0803 generic clearance.

Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, (20 U.S.C. §9573)].

Written consent will be obtained from participants who are over the age of 18 and from parents or legal guardians of students who are under the age of 18. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study and will be destroyed after the final report is completed.

The interviews will be audio-recorded. The only identification included in the audio files will be the participant ID. The recorded files will be secured for the duration of the study – with access limited to key project staff – and will be destroyed after the final report is submitted.

Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.¹ The initial contact and response is estimated at 6 minutes per case for students and 5 minutes per case for teachers and administrators, including follow-up phone calls to screen participants and/or answer any questions that they (or their parents or legal guardians) may have, confirm participation, schedule interviews, and contact participants in advance of the appointment. All interviews will be scheduled for no more than 60 minutes. Table 1 details the estimated burden for the cognitive laboratory testing.

Table 1. Estimate of hourly burden for EDSCLS Cognitive Interviews recruitment and participation

Type	Hours per respondent	Number of respondents	Number of responses	Total hours
Recruitment
Recruiting and contacting students	0.1	64	64	7
Recruiting and contacting teachers and administrators	0.083	64	64	6
Participation
Students (middle school & high school)	1	20*	20	20
Teachers and administrators	1	20*	20	20
Total Burden		128	168	53

* These respondents have already been accounted for under recruitment and thus do not contribute to the total.

Estimate of Costs for Recruiting and Paying Respondents

To encourage their participation, and thank them for their time and effort, all participants will be offered an incentive. Teachers and non-instructional staff will be offered $40 for their participation. Students will be offered $25 for their participation, and parents or legal guardians who bring their student to and from the testing location, will also receive $25 for the time and cost of providing transportation to the student participant. Principals, who are fewer in number and difficult to recruit, will receive $75 for their participation.

Costs to Federal Government

The estimated costs of the cognitive laboratory testing activities in this submittal are $57,771, including the study design and preparation, recruitment, incentive costs, data collection, analysis, and reporting.

Schedule

Recruitment, testing, and reporting will begin in November 2016 and end in February 2017. Recruitment and testing will begin immediately upon OMB approval.

1Assumptions for approximate attrition rates for direct participant recruitment are 50 percent from initial contact to follow-up, 30 percent from follow up to confirmation, and 20 percent from confirmation to participation.

1