Supporting Statement-B

The Safe Schools/Healthy Students (SS/HS) Initiative

National Evaluation

Supporting Statement

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The universe of potential respondents and the sampling method vary for each of the instruments used in data collection for the national evaluation of the SS/HS Initiative. The proposed universe and number of respondents appear in Table 6. The evaluation will collect data either from all possible respondents or from the respondents identified as relevant by the local project director of each grantee; “sampling” per se will not be utilized.

Table 6. Instrument Description and Number of Respondents

Instrument Description	Anticipated Number of Respondents
Site Visit Protocol	100
Baseline Assessment Survey	25
Partnership Inventory	400
Project-Level Survey	100
School-Level Survey	2,300
Staff School Climate Survey	25,200

Site Visit Protocol: Respondent Universe and Sampling Methods. There are an estimated 100 site visits to grantees annually. During the site visit, data are collected from approximately five individuals per site: the project director, the local evaluator, and representatives from the three required partner agencies. Thus, there is a universe of 500 potential and anticipated respondents. No sampling will be done from this universe.

Baseline Assessment Survey: Respondent Universe and Sampling Methods. Respondents for this instrument are approximately 25 project directors or other designees. No sampling will be done from this universe.

Partnership Inventory: Respondent Universe and Sampling Methods. Respondents to this inventory are the representatives from the required partners (average of 4 per each of 100 sites = 400). The total universe of respondents is 340; no sampling will be done from this universe.

Project-Level Survey: Respondent Universe and Sampling Methods. Respondents for this instrument are the approximately 100 project directors. No sampling will be done from this universe.

School-Level Survey: Respondent Universe and Sampling Methods. Respondents for this survey are one selected individual from each school involved in each of the grantee LEAs. The NET anticipates a universe of 2,500 participating schools; no sampling will be done from this universe.

Staff School Climate Survey: Respondent Universe and Sampling Methods. A sample of staff members in all targeted schools will be requested to participate in the Staff School Climate Survey data collection annually. “Staff members” include instructional, administrative, and support staff to ensure that data include a variety of potential perspectives on school climate. In fact, in many smaller school systems, there appears to be no practical distinction between instructional and administrative staff because instructional staff are tasked with administrative responsibilities. Given approximately 100 grantees and 84 responses needed by school type, and three grade level groupings within a given grantee (elementary, middle, and high), the result is the estimate of 25,200 eligible respondents.

A sampling frame that would provide school climate scores representative of the overall grade-level grouping at the .05 level of accuracy would require completed data collection from approximately 76 percent of the staff in each grade-level grouping.

Within the LEAs, the number of targeted schools range from 2 to over 275. Schools are of all various grade-level groupings (e.g., elementary schools, middle schools, high schools).

2. Information Collection Procedures

The LEAs that apply for the SS/HS Initiative grant support are committed by the terms of the SS/HS grant to actively participate in the national evaluation of the SS/HS Initiative. The terms of the grant require preparation of a Memorandum of Understanding with at least three local agencies (juvenile justice system, law enforcement agency, and mental health service provider); the memorandum must include the commitment of these partners to actively participate in the national evaluation. This data collection is characterized as an elaboration of that requirement, and its successful implementation is a true partnership between the grantees and the Federal grantors.

The project director (or designee) of each grantee will be responsible for coordinating data collection activities with the NET. This coordination includes the following responsibilities:

• Participate in the Project-Level Survey annually.

• Identify and recruit a knowledgeable, designated contact for each school receiving programs or services directly or indirectly from the grant, and ensure the designated contact completes the School-Level Survey.

• Inform the principals or other appropriate administrative directors of each school of the critical importance of maximum staff participation in the annual Staff School Climate Survey, and provide updates on the level of participation obtained for each iteration of the survey among the school staff, at least until 75 percent participation is achieved.

• Annually collect and transmit information to the NET on the number of staff members at each school who are eligible to participate in the Staff School Climate Survey to accurately measure completion rates.

• Identify one representative from each partner agency to participate in national evaluation data collection activities including the Site Visit Protocol and Partnership Inventory, and facilitate the participation of the representatives from these agencies in data collection through the Group Interview and the Partnership Inventory and in any activities to clarify information during a site visit or through an individual telephone interview.

Site Visit Protocol: Information Collection Procedures. Since grantees differ on many specific aspects of their grants, the initial site visit protocols serve mainly to guide the data collections. Prior to the site visit, preliminary discussions are conducted to arrange logistics, determine participants, and set a schedule. The initial meeting is with the grant’s project director. The information collected from this meeting is structured by the protocol and recorded (with permission). Occasionally, the project director involves the local evaluator and other individuals who have specific knowledge concerning the grant. The next interview is with the required partners. Again, these interviews are guided discussions but individualized for the specific site. Following the interviews and meetings, discussions are summarized and data are entered into the NET database.

Baseline Assessment Survey: Information Collection Procedures. This survey will be administered online following the postaward call. Respondents will be provided telephone technical assistance if needed and a unique survey link for their grant. Respondents will be given approximately 3 weeks to complete the survey. Followup will be via email.

Partnership Inventory: Information Collection Procedures. This survey will be administered online. Respondents will be provided telephone technical assistance and the unique survey link for their grant. Respondents will be given approximately 4 weeks to complete the survey. Followup will be via email.

Project-Level Survey: Information Collection Procedures. This survey will be administered online. Project directors will be provided telephone technical assistance if needed and a unique survey link. Respondents will be given approximately 4 weeks to complete the survey. Followup with project directors will be via email reminders.

School-Level Survey: Information Collection Procedures. This survey will be administered online. Each participating school will be provided telephone technical assistance and their unique survey login/password. Respondents will be given approximately 4 weeks to complete each survey. Followup with sites on response rates will be via email.

Staff School Climate Survey: Information Collection Procedures. This survey will be administered online through a subcontractor’s Web site. Sites will be provided telephone technical assistance and the survey login/password for each participating school in the sample. Each school will be assigned a unique code number. Based on feedback from the previous administrations of the Staff School Climate Survey, passwords will be generated based on the first four letters of the school name to ensure that staff will not be too burdened using a completely random password.

Each grantee’s project director or a designated liaison will be responsible for identifying eligible respondents and tracking responses by school type to ensure required completion rates.

As lead contractor for the cross-site evaluation, MANILA will provide oversight of its subcontractors and assume responsibility for successful implementation of the data collection.

3. Methods To Maximize Response Rates

Since the site visits are conducted either in person or as a single session, there is no issue regarding response rates.

Baseline Assessment: Methods To Maximize Response Rates. The Baseline Assessment is a critical data collection component as it provides an initial benchmark to compare to future evaluation. The assessment will be administered online to ensure that all recipients receive the link to the Web-based assessment and followup reminders will be sent to nonresponders on a weekly basis. When completion has been delayed beyond 3 weeks from schedule, the site liaison will inform the Federal Project Officer for the grantee and will send an email requesting a scheduled telephone conversation to identify and resolve any barriers or issues that may be delaying response at the site.

Partnership Inventory: Methods To Maximize Response Rates. Completion rates are monitored by a NET staff member. Email reminders will be sent to nonrespondents after approximately 2 weeks. When completion has been delayed beyond 3 weeks from schedule, the NET will inform the grantee project director.

Project-Level Survey: Methods To Maximize Response Rates. Similarly, completion of the Project-Level Survey is an inherent role of the project director of each LEA that receives an SS/HS grant. Completion rates are monitored by the NET site liaisons. When completion has been delayed beyond 3 weeks from schedule, the NET will inform the Federal Project Officer for the grantee and will send an email requesting a scheduled telephone conversation with the project director and the site’s local evaluator to identify and resolve any barriers or issues that may be delaying response at the site.

School-Level Survey: Methods To Maximize Response Rates. A similar process will be in place for identification and resolution of barriers or issues that may be delaying completion of responses by the designated contact at each school responsible for completion of the School-Level Survey. Completion rates are monitored initially by the project director and the NET. When completion has been delayed beyond 3 weeks from schedule, the NET will inform the Federal Project Officer for the grantee and will send an email requesting a scheduled telephone conversation with the project director and the site’s local evaluator to identify and resolve any barriers or issues at the school that may be delaying response.

Staff School Climate Survey: Methods To Maximize Response Rates. The project director for each grantee will contact school staff to encourage participation in the survey and if necessary work with school superintendents and school staff representatives. The NET will also work with the Federal Project Officers overseeing each site to encourage sites to return surveys in a timely manner. As noted earlier, grantees vary from a single isolated tribal school to a county-wide metropolitan school district composed of hundreds of schools and thousands of staff members. A grantee’s structure and context will influence the specific methods used to maximize response rates.

Where an LEA has not achieved at least 60 percent of responses after 3 weeks, the NET will inform the Federal Project Officer for the grantee and will send an email requesting a scheduled telephone conversation with the project director and the site’s local evaluator. The purpose of this phone call is to identify any barriers or issues at the site that may be delaying response. Possible delays might include unexpected school closings, testing or certification schedules, and lack of adequate explanation of the value of completion of the survey to the work of the potential respondents. During the scheduled conversation on barriers to completion, the NET representative and the grantee personnel will reach agreement on locally tailored methods to address the issues that appear to be delaying completion of the data collection effort.

A second progress review of completion rates will be conducted 6 weeks after the launch of the data collection instrument. The target for this milestone will be the expected 75 percent completion rate. Any LEAs reporting less than 75 percent at the 6-week point will receive an email requesting a scheduled telephone conference on how to overcome any barriers inhibiting achievement of the completion rate (see Attachment 8).

A high response rate is anticipated from all schools targeted by the grant because it is believed there will be high recognition of the value of the survey in supporting and sustaining the SS/HS activities after the grant is over. In many LEAs, this value is enhanced by the role of the SS/HS as the first discretionary grant received by the area’s schools.

4. Test of Procedures

Four of the data collection instruments (Site Visit Protocol, Partnership Inventory, Project-Level Survey, and School-Level Survey), developed uniquely for assessment of the planning and implementation processes of the SS/HS grants, were resubmitted to OMB on January 19, 2011. The remaining data collection instrument, the Baseline Assessment Survey, was an addition based on lessons learned from the previous contract. Project directors of previous grantees reviewed the new instrument and four refined instruments to ensure the content, language, and format employed by each instrument were satisfactory. These instruments were further modified to comply with suggestions made by the project directors or to clarify language that participants found problematic. In the event that fine-tuning of the survey instrument is required, OMB will be notified in a memorandum with a copy of the final version of the Web-based survey.

The CHKS, which serves as the basis for the Staff School Climate Survey, has been fully tested for validity and reliability among school staff populations by WestEd under contract with the State of California and has been in use in that State since 2004. Several other States also have adopted the survey, and the NET has been administering the Staff School Climate Survey under the current contract since 2009.

The latent unidimensionality of the subscales generated from the Staff School Climate Survey items has been tested using Cronbach’s alpha, a coefficient of reliability or consistency. When the average interitem correlation is low, alpha is low. As the average interitem correlation increases, Cronbach’s alpha also increases. By implication, when interitem correlations are high, items are measuring the same underlying construct, and reliability is “high” or “good.”

In the case of the subscales of the Staff School Climate Survey, the three-item scale of perceptions of health and counseling services received the lowest alpha: 0.70. However, this is considered acceptable interitem reliability. All the other subscales showed good reliability, with alphas ranging from 0.85 to 0.95. The instrument will use these subscales to track changes in school climate as measured by changes in the aggregate perception of members of the school staff in schools receiving benefits of targeted Federal funding from the SS/HS Initiative.

5. Statistical Consultants

The primary contractor MANILA and its subcontractors Battelle Centers for Public Health Research and Evaluation and RMC will have overall responsibility for implementation and execution of the project, including data collection and analysis.

The primary contractor for this project is:

MANILA Consulting Group, Inc.

Gary Hill, Project Director

1420 Beverly Road, Suite 220

McLean, VA 22101

571-633-9797, ext. 208

[email protected] subcontractors for this project are:

Battelle Centers for Public Health Research and Evaluation

Ping Yu, Subcontract Manager

2111 Wilson Boulevard, Suite 900

Arlington, VA 22201

703-875-2981

[email protected]

RMC Research Corporation

Steve Murray, Subcontract Manager

111 SW Columbia Street, Suite 1200

Portland, OR 97201

[email protected]

800-788-1887

The Federal Government Project Officer is:

Nainan Thomas, M.S.W., LL.B., Ph.D.

Center for Mental Health Services

Substance Abuse and Mental Health Services Administration

One Choke Cherry Road, Room 6-1099

Rockville, MD 20857

240-276-174

[email protected]

List of Attachments

Attachment 1: Letter of Introduction of the SS/HS Cross-Site Evaluation

Attachment 2: Data Collection Instrument—Baseline Assessment

Attachment 3: Data Collection Instrument—Site Visit Protocol

Attachment 4: Data Collection Instrument—Partnership Inventory

Attachment 5: Data Collection Instrument—Project-Level Survey

Attachment 6: Data Collection Instrument­­—School-Level Survey

Attachment 7: Data Collection Instrument—Staff School Climate Survey

Attachment 8: Sample Emails for Instruments

Attachment 9: Data Collection Instrument—GPRA Student Survey

8