Ssb

“Assessing Education Agency Staff Perceptions of School Climate and Youth Access to Services”

OMB #0920-1048

Expiration Date: 2/28/2018

Supporting Statement Part B

Revision

January 26, 2018

Supported by:

Centers for Disease Control and Prevention

Catherine Rasberry, PhD

CDC/OID/NCHHSTP, Health Scientist

(404) 718-8170 (phone)

(404)-718-8045 (fax)

[email protected]

B.1 Respondent Universe and Sampling Methods

B.2 Procedures for the Collection of Information

B.3 Methods to Maximize Response Rates and Deal with No Response

B.4 Tests of Procedures or Methods to be Undertaken

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Section B: Collections of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

Web-based Instrument

The respondent universe for the Web-based instrument will consist of selected staff in 3 local education authorities (LEAs) that are participating in strategy 4 of PS13-1308: Broward County Public Schools (BCPS) in Broward County, Florida; Los Angeles Unified School District (LAUSD) in Los Angeles, California; and San Francisco Unified School District (SFUSD) in San Francisco, California. Seven high schools are participating in each of the 3 funded LEAs, and up to 35 staff members will be asked to participate per school, for a total of up to 735 respondents across the 3 LEAs. (The 3 LEAs funded under the cooperative agreement have chosen to focus program activities in high schools—not junior high schools—and as a result, the proposed data collection focuses on collecting information from staff of high schools.) Data collection covered under the previously approved ICR was originally administered during the 2014-2015 and 2016-2017 school year. These dates coincide with the beginning and middle of program activities. The current ICR will cover data collection during the 2017-2018 school year and will serve as the final data collection, corresponding with the end of the CDC-funded program activities. The samples will be treated independently; the study is not longitudinal.

CDC’s evaluation contractor will work with the principal (or other administrator) of each school to determine which school staff should be included in the Web-based data collection, based on their job role/responsibilities and the relevance of their work to the study questions the assessment seeks to answer. The anticipated mix of invited participants from each school at each time point is outlined in the table below

Respondents	Number
Principals and other administrators	3
Counselors	3
Nurses	2
English teachers	15
Arts teachers	5
Key informants (club sponsors, health, other staff)	7
Total per school	35
TOTAL RESPONDENT UNIVERSE AT EACH ADMINISTRATION (21 Schools)	245 Respondents per district 735 Respondents Total

This sampling approach is consistent with that used for data collection under the previously approved ICR for this study.

Interview Guide

The respondent universe for the interviews consists of up to 44 BCPS employees: 2 district-level administrators, 14 school-level administrators (2 from each of 7 schools), and 28 (4 staff from each of 7 schools) school staff from the 7 participating schools. This study population includes individuals employed in the following positions: principals, assistant/ or vice principals; guidance counselors; school nurses; physical education coaches; teachers; and 1-2 individuals employed in administrative positions responsible for policy and student services.

A purposive sampling technique will help identify school staff who can answer questions about policies and activities related to maintaining safe and supportive school environments. We will work with the BCPS Office of Diversity, Cultural Outreach and Prevention to identify district-level administrators to participate in interviews. We will also ask BCPS to provide a list of principals for each school, and provide each school principal with guidance on how to identify school staff who can describe implementation of relevant policies and activities.

Other than the recommendation and/or approval for inclusion by the Project Coordinator at BCPS or individual school principal, the criteria for inclusion will be:

• Being currently employed by Broward County Public Schools (the Broward County LEA), either at the district level or at one of the high schools selected by BCPS for participation in the CDC-funded project (BCPS has chosen to focus program activities in high schools—not junior high schools—and as a result, the proposed data collection focuses on collecting information from staff of high schools);

• Having worked in their current position for at least one year. This is to ensure that site informants have had a period of time during which to become familiar with district and/or school policies, programs and environment.

This sampling approach is consistent with that used for data collection under the previously approved ICR for this study.

B.2 Procedures for the Collection of Information

Web-based Instrument

The Web-based instrument (see Attachments 3, 4, and 5 for the MS Word version of the information collection instruments and Attachments 6, 7, and 8 for the Web version of the information collection instruments) will be administered in the 2017-2018 school year. The Web-based instruments will be distributed, and data will be collected, using the Web-based data collection tool, SurveyMonkey®. Staff will access the instrument using a link sent via email. The assessment will take 20-25 minutes to complete, and staff will be permitted to take it on their own computer or a school computer at a convenient time (during the school day or outside of school hours), as determined in conjunction with the LEA and school administrators. The decision about specific administration dates is made in coordination with each school to ensure that the time of administration selected results in the least burden/highest possible acceptability for the school. Up to 3 reminder emails will be sent out over the 2 week period. Once data collection is complete, all MS Excel files downloaded from SurveyMonkey® will be converted to an SAS or SPSS file format for analysis.

Interview Guide

The interviews will be conducted in the 2017-2018 school year during site visits to the seven participating schools in BCPS. Comparison of this data to data collected under the previously approved ICR will be used to assess (qualitatively) if any change related to creating a safe and supportive environment has taken place in the participating schools. Interviews will be conducted using the interview guide (see Attachment 9: School Climate Index Interview Guide for District-level Administrators; Attachment 10: School Climate Index Interview Guide for School-level Administrators; and Attachment 11: School Climate Index Interview Guide for School Staff) and will be conducted either individually or in small groups. Teams of trained interviewers will work in pairs to conduct up to 6 interviews at each school and two interviews at the BCPS district office. The evaluation contractor will make arrangements to conduct the interviews in a quiet place that provides privacy (or over the phone, if the participant is not available during the site visit). Each interview will last 60-90 minutes and will be audio-recorded and, with permission, transcribed. Transcripts will be loaded into Atlas.ti, MAXQDA, or similar qualitative data analysis software for analysis.

Power Analysis

Web-based Instrument Power Analysis

We have powered this study to detect a pivotal outcome of the program, increasing the rate of referral to existing community services by school staff members. Currently, referral rates within each district are minimal, and we hypothesize a 10% change in referral rates among staff members across all districts from the initial to the final data collection. For the analysis, we will analyze data pooled across all school districts. Assuming a response rate similar to that achieved at baseline (similar to other teacher surveys without incentives, we approached a 50% response rate), our total sample size of 642 staff across all districts will give our study a power = .94 when comparing proportions and power = .97 when comparing means. Knowing that other outcomes of interest will have greater effect sizes, we have powered the study on referral rates with the confidence we will have even greater power to detect change in other outcomes of interest for school staff members.

Number of staff at baseline	321 Staff
Estimated number of staff at the final data collection	321 Staff
Total Respondents	706 Staff Power (proportion) = .94 Power (mean) = .97
Notes: Questionnaire will have a maximum sample size per LEA of 490. Power assumes detection of 10% difference, two tailed test. Power of mean comparison assumes a small effect size (.15). Power assumes independent samples in baseline and final data collections.

Figure1. Power estimate for detecting a 10% difference in the referral rate of youth to testing facilities by school staff across all districts (two tailed test)

Figure 2. Power estimates assuming a small effect size (f=.15) to detect a difference in mean referral rates of youth to testing facilities by school staff across all districts

Interview Guide Power Analysis

The interview-based assessment uses a purposive sample drawn from staff from seven BCPS high schools serving as project sites and the BCPS district office. The selection of respondents will be conducted to ensure the most appropriate individuals are identified and invited to participate. Using the description of the School Climate Assessment provided in the introductory email, the principals can nominate a variety of staff types for inclusion in the interviews to ensure broad input and reduce potential for bias. However, for this qualitative data collection, we do not expect that the results will be generalizable to school or district staff. Although we will examine the data for changes in themes, we will not track changes in individual responses over time; the study is not longitudinal.

B.3 Methods to Maximize Response Rates and Deal with No Response

Participating schools have signed letters of agreement to participate in the prevention project and related evaluation activities, including facilitating recruitment for the Web-based survey and school climate assessment interviews. Specific efforts to promote participation in each of the data collection activities are described below.

Web-based Instrument

School staff will have a 2-week window during which to complete the assessment. Staff will be permitted to take it on a school computer at a convenient time (during the school day or outside of school hours), as determined in conjunction with the LEA and school administrators. CDC and CDC’s evaluation contractor will work with each school administrator to determine the timing of the web-based assessment to minimize disruption to staff and maximize the opportunity for participation.

To encourage participation, contractor data collectors will work with school administrators throughout the process to ensure recruitment of the correct staff, and send up to 3 reminders to ensure completion of the web-based assessment. School administrators may also inform staff to expect being contacted about the assessment, either by sending an introductory email telling staff to expect an email about the instrument or by mentioning the assessment in a staff meeting. This recruitment process will be the same as it has been in previous years.

Interview Guide

The semi-structured interview is brief (60 to 90 minutes), and was designed for school and district-level employees. Interviews will be scheduled at a time that is convenient for each staff member or group of staff members. Interview slots before, during and after school hours will be offered. The contractor will make arrangements before the interview to conduct the interview in a quiet place that provides privacy. Interview candidates will be informed of measures that are in place to protect their identities; no information that could link a participants’ name to his/her interview will be released to anyone outside of the project team (project director, project managers, and interviewers).

B.4 Tests of Procedures or Methods to be Undertaken

Web-based Instrument

The Web-based instrument was developed for this study through an iterative process of reviewing existing school staff instruments, and identifying gaps in the inventory of available items to be filled by study-created questions. Whenever possible, items were selected from valid and reliable data collection instruments. The staff questionnaire was piloted by 5 individuals, 3 of whom were teachers or former teachers, and 2 of whom were members of the evaluation team. Participants in the pilot were asked to provide input on the questionnaire content and also duration of administration. Refinements were made to the instrument based on feedback and the range of time to complete the questionnaire was 20 to 25 minutes. The refined questionnaire was used effectively and as intended for data collection under the previously approved ICR.

Interview Guide

The semi-structured interview guides were developed with the extensive input of expert consultants both internal and external to CDC and the evaluation contractor. The interview guides were adapted from instruments developed by our consultant for this project, Hetrick-Martin Institute (HMI). HMI has used the original tools to conduct informal assessments with schools in New Jersey and New York. Both before, and subsequent to adaptation, interview guides and procedures were reviewed by both the evaluation contractor and CDC. We piloted the adapted instruments for duration of administration and refinement with 3 individuals with evaluation and/or school administration experience and used their feedback to refine the instruments and prepare for interviewer training. The refined guide was used effectively and as intended for data collection under the previously approved ICR.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Individuals consulted on statistical aspects and study design:

Catherine Lesesne, PhD

Technical Director

ICF International

3 Corporate Square, Suite 370

Atlanta, GA 30329

[email protected]

404-321-3211 (phone)

Andrew Hebert, MPH

Data Manager Associate (former)

ICF International

3 Corporate Square, Suite 370

Atlanta, GA 30329

404-321-3211 (phone)

The individuals overseeing data collection are:

Catherine Lesesne, PhD

Technical Director

ICF International

3 Corporate Square, Suite 370

Atlanta, GA 30329

[email protected]

404-321-3211 (phone)

Elizabeth Kroupa, MPH

Manager

ICF International

710 Second Ave, Suite 550

Seattle, WA 98104

[email protected]

206-801-2800 (phone)

Catherine Rasberry, PhD

CDC/OID/NCHHSTP, Health Scientist (Contracting Officer’s Representative providing oversight)

(404) 718-8170

[email protected]

The individuals directing data analysis are:

Catherine Lesesne, PhD

Technical Director

ICF International

3 Corporate Square, Suite 370

Atlanta, GA 30329

[email protected]

404-321-3211 (phone)

Elizabeth Kroupa, MPH

Manager

ICF International

710 Second Ave, Suite 550

Seattle, WA 98104

[email protected]

206-801-2800 (phone)

Catherine Rasberry, PhD

CDC/OID/NCHHSTP, Health Scientist (Contracting Officer’s Representative providing oversight)

(404) 718-8170

[email protected]

10