Justification

Background and Study Rationale

The ED School Climate Surveys (EDSCLS) are a suite of survey instruments being developed for schools, school districts, and states by the U.S. Department of Education’s National Center for Education Statistics (NCES). This national effort extends current activities that measure school climate, including the state-level efforts of Safe and Supportive Schools (S3) grantees, which were awarded funds in 2010 by the Department of Education’s Office of Safe and Healthy Students (OSHS) to improve school climate. Through the EDSCLS, schools nationwide will have access to survey instruments and a survey platform that will allow for the collection and reporting of school climate data across stakeholders at the local level. The surveys can be used to produce school-, district- and state-level scores on various indicators of school climate from the perspectives of students, teachers and staff, principals, and parents and guardians. NCES will also provide benchmark data, collected from a nationally representative sample of schools across the United States, to facilitate comparisons between school climate scores at the local and national levels (OMB# 1850-0923 v.1).

As part of EDSCLS development process, NCES conducted cognitive interviews in 2014 (OMB# 1850-0803 v.102 and 107) and a pilot test in 2015 (OMB# 1850-0803 v.119 and 129). In 2015, the Office of Safe and Healthy Students (OSHS) requested that new items addressing school climate and sexual assault be added to the EDSCLS questionnaires. School communities nationwide have identified sexual assault as an issue that students and schools are increasingly facing. OSHS has requested for EDSCLS to produce data to assist schools in creating supports and protective factors for reducing sexual assault as part of creating a safe and healthy learning environment. This submission requests to cognitively test the new items in order to determine whether they can be included in the EDSCLS survey and how should they be worded.

Type of Testing

We will conduct cognitive interviews to test the new items developed for the EDSCLS questionnaires. Using a paper survey, the interviewer will apply a structured protocol in a one-on-one interview drawing on methods from cognitive science. The cognitive interviews will investigate the cognitive processes that respondents use to answer survey questions. In particular, these interviews will identify problems of ambiguity or misunderstanding in question wording. We are particularly intent on ensuring that all items included in the final surveys are easily understood by respondents, with their interpretations consistently aligned with the concepts being measured.

We plan to use think-aloud interviewing and verbal probing techniques (also known as concurrent and retrospective recall probing, respectively). With think-aloud interviewing, respondents are explicitly instructed to think aloud (i.e., describe what they are thinking) as they work through items. With verbal probing techniques, the interviewer asks probing questions, as necessary, to clarify points that are not evident from the “think-aloud” process, or to explore additional issues that have been identified a priori as being of particular interest. Cognitive interview studies produce qualitative data in the form of verbalizations made by participants during the think-aloud phase and in response to the interviewer probes.

Survey Items

The student cognitive interviews will include 10 items: two items on sexual assault and, to provide context, eight items that on the survey instrument are adjacent to the sexual assault items.

The student item Sengrel153, “At this school, there is a teacher or some other adult who students can go to if they need help because of sexual assault or dating violence,” performed well in the EDSCLS high school student survey pilot test but has not been tested using cognitive interviews. The item has also not been tested using middle grade students

The student item Ssafpsaf154, “Students at this school feel unsafe because of sexual assault or dating violence,” has not been previously tested.

The instructional and non-instructional staff cognitive interviews will include 9 items: one item on sexual assault and, to provide context, eight items that on the survey instrument are adjacent to the sexual assault item.

The staff item Isafpsaf143, “The following types of problems occur at this school often: sexual assault or violence.” Has not been previously tested.

Sample and Recruitment Plans

Research Support Services (RSS), under subcontract to RTI International (RTI) will conduct the cognitive laboratory testing in October and November 2015. Participants will include the target recipients of each of the four questionnaires: middle school students, high school students, teachers and instructional staff, and principals and non-instructional staff. They will be recruited from the Chicago metropolitan area where the RSS office is located. Recruitment will continue until interviews are completed with 10 participants from each group. We anticipate recruiting about 32 potential participants per group to achieve the 10 completed interviews in each.

RSS will attempt to recruit participants representing a range of characteristics (including urban and suburban areas, students from a mix of grades, and staff serving in a variety of school roles). We will work to ensure that both principals and professional staff are included in the sample for testing the principal/non-instructional staff questionnaire. Please note that although the sample will include a mix of characteristics, the results will not explicitly measure differences by these characteristics.

We will use multiple outreach methods and resources, such as marketing research companies, newspaper/internet ads, and contacts with community organizations (e.g., libraries and afterschool programs) to recruit participants. Paper flyers, e-mails, or phone calls will be used to contact the potential participants. Interested participants will be screened using a screener script to ensure they meet the criteria for participation in the interviews. During this communication, the parent or guardian of any interested minors will be informed about the objectives, purpose, and participation requirements of the data collection effort, as well as the activities that it entails. Only after RSS has obtained written consent from the parent or guardian will the student be allowed to participate in the cognitive laboratory testing session. RSS will confirm the interview date, time, and location via e-mail, letter, or telephone. Principals, non-instructional staff, teachers, students, and the parents or legal guardians of participating students will complete consent forms at the time of the interview. We will continue recruiting participants until the goal of ten completed interviews is reached for each of the participant groups.

The cognitive laboratory testing will take place in a space rented by RSS or at a local library convenient to participants. Participants will first be welcomed, introduced to the interviewer and the observer (if an in-room observer is present), and told that they are there to help answer questions about how people answer school climate survey items. All participants will be reassured that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Interviewers will explain the think-aloud process and conduct a practice session with a sample question.

The think-aloud component of the cognitive interviews will use either (1) a concurrent think-aloud method in which the participant verbalizes his or her thoughts while working through a questionnaire or (2) a retrospective think-aloud method during which participants work through a questionnaire silently and then discuss their thoughts about the item content while working through it a second time. The cognitive interviews will also include a verbal probing component conducted after completion of the think-aloud portion for a given questionnaire. The verbal probes will include a combination of pre-planned item-specific questions, identified before the session as important and ad hoc questions that the interviewer identifies as important from observations during the interview, such as clarifications or expansions on points raised by the participant. To minimize the burden on the participant, efforts will be made to limit the number of verbal probes that can be used in any one session. The protocols will contain generic prompts to be applied flexibly by the interviewer to facilitate and encourage participants to verbalize their thoughts. For example: “I see you’re looking at the answer. What are you thinking?” or “Are there any questions or words that seem confusing here?” Observers will take notes on what participants say and the sessions will be audio recorded.

The results will be compiled to identify patterns of responses and any item specific issues. This approach will help to ensure that the data are analyzed in a thorough and systematic way that enhances the identification of problems with items or tasks and provides recommendations for addressing them.

Assurance of Confidentiality

Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, (20 U.S.C. §9573)].

Written consent will be obtained from participants who are over the age of 18 and from parents or legal guardians of students who are under the age of 18. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files and secured for the duration of the study and will be destroyed after the final report is completed.

The interviews will be audio-recorded. The only identification included in the audio files will be the participant ID. The recorded files will be secured for the duration of the study – with access limited to key project staff – and will be destroyed after the final report is submitted.

Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.¹ The initial contact and response is estimated at 6 minutes per case for students and 5 minutes per case for teachers and administrators, including follow-up phone calls to screen participants and/or answer any questions that they (or their parents or legal guardians) may have, confirm participation, schedule interviews, and contact participants in advance of the appointment. All interviews will be scheduled for no more than 60 minutes. Table 1 details the estimated burden for the cognitive laboratory testing.

Table 1. Estimate of hourly burden for EDSCLS Cognitive Interviews recruitment and participation

Type	Hours per respondent	Number of respondents	Number of responses	Total hours
Recruitment
Recruiting and contacting students	0.1	64	64	7
Recruiting and contacting teachers and administrators	0.083	64	64	6
Participation
Students (middle school & high school)	1	20*	20	20
Teachers and administrators	1	20*	20	20
Total Burden		128	168	53

* These respondents have already been accounted for under recruitment and thus do not contribute to the total.

Estimate of Costs for Recruiting and Paying Respondents

To encourage their participation, and thank them for their time and effort, all participants will be offered an incentive. Teachers and staff will be offered $40 for their participation. Students will be offered $25 for their participation, and parents or legal guardians who bring their student to and from the testing location, will also receive $25 for the time and cost of providing transportation to the student participant.

Costs to Federal Government

The estimated costs of the cognitive laboratory testing activities in this submittal are $55,869, including the study design and preparation, recruitment, incentive costs, data collection, analysis, and reporting.

Schedule

Recruitment, testing, and reporting will begin in October and end by early November 2015. Recruitment and testing will begin immediately upon OMB approval.

1Assumptions for approximate attrition rates for direct participant recruitment are 50 percent from initial contact to follow-up, 30 percent from follow up to confirmation, and 20 percent from confirmation to participation.

1