The Safe Schools/Healthy Students (SS/HS) Initiative
National Evaluation
Supporting Statement
A. Justification
1. Circumstances of Information Collection
The Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Mental Health Services (CMHS) is seeking Office of Management and Budget (OMB) approval for a revision (OMB No. 0930-0297). The following seven instruments are used to collect information for conducting the national evaluation of the Safe Schools/Healthy Students (SS/HS) Initiative:
Year 1 Site Visit protocol
Group Interview protocol
Project Director Interview protocol
Partnership Inventory
Project-Level Survey
School-Level Survey
Staff School Climate Survey
SAMHSA awarded a new contract to continue the national evaluation on September 29, 2010. This contract modifies the scope and nature of the data collection activities. These changes require revisions to the seven instruments currently authorized under OMB No. 0930-0297. These changes have resulted in the reconfiguration of information collection using the following six instruments:
Baseline Assessment Survey
Site Visit Protocol
Partnership Inventory
Project-Level Survey
School-Level Survey
Staff School Climate Survey
The SS/HS Initiative is authorized under the Safe and Drug-Free Schools and Communities Act (20 U.S. Code [U.S.C.] 7131), Public Health Service Act (42 U.S.C. 290[hh]), and Juvenile Justice and Delinquency Prevention Act (42 U.S.C. 5614[b][4][e] and 5781 et seq.). The initiative is an unprecedented collaborative grant program supported by three Federal departments—the U.S. Departments of Health and Human Services, Education, and Justice.
The SS/HS Initiative provides funds for grantees—Local Education Agencies (LEAs)—to use, and sometimes develop, state-of-the-art knowledge about what programs and practices work best to foster resilience; promote safe and healthy environments where America’s children can learn and develop; and prevent violence and substance use among our Nation’s youth, schools, and communities. Contributing to the uniqueness of this initiative is the requirement that each grantee include partnership collaboration among the LEAs and representatives from local law enforcement, mental health services, and juvenile justice agencies.
The national evaluation is conducted under the authority of the Secretary of Health and Human Services and is mandated under 42 U.S.C. 290(hh), item (f):
The Secretary shall conduct an evaluation of each project carried out under this section and shall disseminate the results of such evaluations to appropriate public and private entities.
The Federal Evaluation Workgroup for the SS/HS Initiative1 determined that LEAs receiving the grant must meet the requirements of the Government Performance and Results Act (GPRA) of 1993 (P.L. 103–62). GPRA requires all Federal agencies to set program performance targets and report annually on the degree to which the previous year’s targets were met. Agencies are expected to evaluate their programs regularly and to use results of these evaluations to explain their successes and failures and justify requests for funding. To meet the GPRA requirements, SAMHSA must collect performance data (i.e., GPRA data) from grantees (OMB No.1894-0003, expiring February 28, 2011).
In addition to these required outcome data, the Federal Evaluation Workgroup determined that this mandate requires collection of detailed process information on planning and implementation of grant activities at each grantee site, particularly with respect to the collaborative partnerships. To ensure the greatest usefulness and generalizability of the national program evaluation, it is critical to determine the factors leading to both positive and negative results. A careful examination of the following will enable clear communication to the public of the effectiveness of the initiative:
Conditions existing at the onset of projects
The impact of the partnerships
How and what programs, activities, and services were selected and implemented
The impact of a wide variety of potential intervening events and intermediate outcomes
2. Purpose and Use of Information
The Federal agencies collaborating on the initiative share an expectation that LEAs and communities nationwide will benefit from the documented experiences of the grantees. This has been made evident in cases where data from the national evaluation were used to inform programmatic changes. For example, based on data from interviews with grantees, the program began requiring grantees to form Core Management Teams as a better way to organize the local project and achieve improved outcomes. Based on a review and analysis of the GPRA data, the data requirements were adjusted to be clearer and provide more standardized data across grantees, resulting in improved data not only for the national evaluation but also for the Department of Education and SAMHSA.
Grantees themselves are able to use national evaluation data. For example, some use data from the Staff School Climate Survey to make comparisons of behavioral data from youth surveys and previous years’ school climate data to examine trends. Some have used the data to serve as project-level indicators and for setting or reassessing objectives, benchmarks, or goals in their local evaluation. More specifically, some grantees have used the data to identify areas of need for specific topics or types of programming. For example, one site had several schools request information on bullying based on information from the results of the School Climate Survey. Another site’s superintendent mandated that schools hold regularly scheduled professional development sessions to address the issues.
After receiving a notice of award, letters introducing the National Evaluation Team (NET) are mailed to all grantees from the director of the Division of Prevention, Traumatic Stress and Special Programs at SAMHSA (see Attachment 1). The NET developed instruments and received approval to collect detailed quantitative and qualitative data on the grant activities and the local collaborative process. These instruments apply to the local project director of the grant, the local project evaluator, one representative of each local organization that formally partners in the administration of the grant activities, one representative of each school receiving services through this grant, and instructional and administrative staff at targeted schools. These instruments included:
A Site Visit Protocol administered during the first year of the grant
Group telephone interviews with project leadership at each grantee site
Project Director Interviews at each grantee site
A Partnership Inventory, completed by a representative of local grant partners
Project-Level Survey completed by project directors
School-Level Survey completed by a representative of each participating school (see Attachment 6)
Staff School Climate Survey completed by staff at all participating schools (see Attachment 7)
The new contract includes the collection of baseline information immediately after the notification of award and annual site visits. These new requirements necessitate the following changes to the current set of approved instruments to improve the quality of the information collected:
Administration of a new Baseline Assessment Survey (see Attachment 2)
Revision and consolidation of the existing Site Visit Protocol, the Group Telephone and Project Director Interview into a new Site Visit Protocol (see Attachment 3)
Revision to the Partnership Inventory (see Attachment 4)
Revision to the Project-Level Survey (see Attachment 5)
Revision to the School-Level Survey (see Attachment 6)
Staff School Climate Survey (see Attachment 7)
Previously, there was no Baseline Assessment instrument. Because of limitations of certain data collection instruments, and in response to feedback from the External Workgroup, the NET proposed adding a baseline assessment instrument to administer to the project director or other key staff member after grant award. The NET previously relied on public demographic data and interview data for most of this information, which was not uniform or available across all sites. The new Baseline Assessment Survey will provide a common baseline for cross-site evaluation and a comparison point for information collected in other surveys, such as the Project-Level Survey. The burden on respondents is estimated to be 40 minutes. The new instrument appears in Attachment 2.
The Site Visit, the Group Telephone Interview, and the Project Director Interview have been revised and consolidated into a new Site Visit Protocol for the anticipated 1½-day site visits (9 hours of interview time). Within each of the interviews, there are several changes and revisions to streamline the protocols, including deletions of items that were not useful and the addition of new items to obtain information to support hypotheses in the Program Theory Model. Changes to these instruments also reflect the requirements in the new Statement of Work that requires site visits instead of telephone interviews in the grant’s out-years. In the revised Site Visit Protocol, there are 34 new items added to and 8 items removed from the earlier version. Despite the net addition of items, the burden on grantees should be reduced by 3 hours with the consolidation of the instruments. The revised protocol appears in Attachment 3.
The Partnership Inventory previously was broken into a project director version and a required partner version; for the proposed inventory, only partners will complete the inventory, reducing the burden on project directors. One component of the Partnership Inventory, the 10-item, self-evaluation of partner contribution, has been replaced with a consolidated 5-item evaluation. There are 31 short new items. With the revisions, the time burden (15 minutes) remains the same for respondents but will yield more useful data. The revised instrument appears in Attachment 4.
The proposed Project-Level Survey contains 102 items; 69 of these are original or slightly revised from the previous version, and 33 are new. Seventeen of the new items are activities or programs that replace activities and programs not consistent with the new Request for Application. This new list of activities and programs contains 39 items and is the same list used in the Baseline Assessment Survey and School-Level Survey. This will facilitate comparisons across instruments. A total of 31 items were removed, most of which were programs and activities not consistent with the new Request for Application. The revisions and changes to this instrument are aimed at collecting only relevant data. With the net reduction of eight survey items, the burden on respondents is approximately 25 minutes. The revised instrument appears in Attachment 5.
The School-Level Survey consists of 90 items; 43 of these are original or slightly revised from the previous version of the instrument. There are 8 new items and a reduction of previous items in a list of programs and activities from 77 to 39 to be consistent with other instruments (the Baseline Assessment Survey and the Project-Level Survey) and more aligned with grantee implementation patterns. Another notable change is modification of response options in the list of programs and activities to facilitate collection of more detail on the level of implementation at the school. An additional 14 items were removed from the survey. The revised survey has 43 fewer items than the previous version, resulting in a reduction of time needed to complete the survey (burden of 25 minutes). The revised instrument appears in Attachment 6.
These instruments, with the exception of the Staff School Climate Survey, collect information concerning process components of grants. The Staff School Climate Survey provides additional outcome information that extends beyond required GPRA outcome measures and the estimated time remains the same at 7 minutes.
2a. Evaluation Overview
An integral part of the SS/HS Initiative is process evaluation information that is currently collected annually via surveys and interviews of grantee and LEA representatives. Instituted by Congress following the murderous assaults at Columbine High School in Colorado, the SS/HS Initiative is designed to provide LEAs, including school districts and multidistrict regional consortia, with grant funding for up to 4 years to simultaneously address activities in the following areas:
Safe school environments and violence prevention activities
Alcohol and other drug prevention activities
Student behavioral, social, and emotional supports
Mental health services
Early childhood social and emotional learning
The specific activities to be conducted at each site and the mode and means of interagency collaboration and partnership at the local level are at the discretion of each grantee. However, senior-level representatives from the local law enforcement, mental health services, and juvenile justice agencies are “required partners” for each grant.
2b. Evaluation Instruments
1. Baseline Assessment Survey. This is a new survey instrument that is designed to collect information about conditions describing the school district(s), partnership, and community prior to grant award. The survey is Web-based and asks specific questions about:
Partnership history
Community characteristics
Existing resources
Pre-grant planning
Existing programs and activities, including evidence-based programs
Coordination and service integration activities
This information provides a comparison point for information collected in other surveys, such as the Project-Level Survey. Currently, most of this baseline information is not collected until the initial site visit, which is typically scheduled 8 or 9 months after the grant award. The survey will be completed by the local project director or other key staff person immediately after the postaward call.
2. Revised Site Visit Protocol: Purpose and Use of Information. Information provided by grantees in their grant application will be organized with the help of a protocol for a site visit to be conducted shortly after award of the grant and annually thereafter. Specific content of questions during the site visit will vary, depending on the content and comprehensiveness of the grant application. The protocol will provide a comprehensive set of subtopics presented in question form in two parts and seven topical areas:
Part 1. Project Director/Local Evaluator
Planning for the SS/HS project
Partnership
Current status of project implementation
Local evaluation status
Part 2. SS/HS Core Management Team/Partnership
Partnership history/update
Enhanced interagency service systems and structures
Sustainability
The project director/local evaluator portion of the protocol will be used primarily to assess each partner’s contribution to the core elements of collaborative functioning and to:
Update information about the programs, strategies, and activities the sites intend to implement
Secure information regarding the site’s perspective on the impact of the SS/HS project on students, families, and the community
Identify local evaluation activities
This information will be used to refine project classifications, examine changes in the number and types of evidence-based practices being implemented, and document the number and type of new service structures or systems that sites plan to implement through the grant.
The group interview portion of the protocol is designed to assess the status of the following:
The SS/HS project as a whole
Structure and functioning of the community partnership
Partners’ involvement in the project
Efforts to enhance service integration and systems change
Perceived impact of the SS/HS project
Since these data reflect different perspectives of participants, information from the various sources must be collected, analyzed, and synthesized. Key informants will include (in addition to the project director) the local evaluator, required partners from each site, and representatives from other key partner organizations that are involved in planning and decisionmaking for key areas of the grant (e.g., alcohol and drug prevention or treatment agencies, after-school programs, early childhood programs). The NET will consult with the local project directors (and Federal Project Officers as necessary) in deciding which partner organizations will serve as key informants during the site visits. Since these data represent local grant conditions and ongoing partner activities, they cannot be obtained from existing sources or other instruments and will be used to address the following central evaluation questions:
How do existing conditions and resources in the pre-grant environment moderate the relationships among grant operations and outcomes?
What SS/HS grant operation characteristics are associated with improvements in near-term and long-term outcomes?
3. Revised Partnership Inventory: Purpose and Use of Information. The Partnership Inventory is a Web-based questionnaire completed by designated representatives of local partnering organizations. Its purpose is to obtain perceptions of operating characteristics of the partnership through Likert-type scaling. Respondents are asked to give their opinions about how the SS/HS partnership is functioning in their community and the interactions among the participating members of the collaboration. These data are used primarily to understand partner perceptions of the local partnership’s functioning level
What SS/HS grant operation characteristics are associated with improvements in near-term and long-term outcomes? What near-term outcomes are associated with improvements in long-term outcomes? Overall, does the SS/HS Initiative meet the Federal Government’s expectations of achieving improvements in long-term outcomes (reduction in substance use and violence, increased access to mental health services, and improvement in school climate) and near-term outcomes (comprehensive programs and activities and improved coordination and service integration)?
5. Revised School-Level Survey: Purpose and Use of Information. The School-Level Survey, completed by a staff person in each targeted school (often an SS/HS coordinator or principal) is designed specifically to provide an indicator as to whether and how project-level SS/HS-related programs, activities, and services are diffused to the individual targeted schools. The survey solicits information on the school’s efforts relating to the SS/HS Initiative since grant award.
Specific questions addressed in this survey include:
Which SS/HS Initiative activities are implemented in the school?
Who is involved in the implementation process?
What is the perceived impact of the initiative at the school level in the different grant areas?
2c. Summary of Evaluation Information
Core data expectations for the evaluation are presented in Table 1. The evaluation questions identified are presented in the column on the left. The specific survey instruments and/or interview protocols that address these questions appear in the column on the right.
Prior to fielding the Baseline Assessment, Partnership Inventory, Project-Level, School-Level, and Staff School Climate Surveys, an email and/or letter will be sent to project directors and other relevant stakeholders to explain the purpose of the surveys and provide information on how to complete them. The email will provide names, email addresses, telephone numbers, and fax numbers for the NET contact(s) to ensure respondents have contact information if they need to clarify survey-related questions. Grantees will also be given instructions if they prefer or need to complete and return hardcopy versions of the survey.
All respondents will receive weekly email reminders about the surveys to ensure high response rates in a timely fashion. The NET also plans additional followup efforts to track any respondents who fail to submit their completed surveys after the initial followup. Sample emails for these instruments are included in Attachment 8.
2d. School Climate Overview
In addition to the instruments for the evaluation described above, the Staff School Climate Survey that measures particular aspects of project outcomes will be administered annually to instructional and administrative staff at targeted schools of each LEA that has been awarded an SS/HS Initiative grant.
The Federal Evaluation Workgroup determined that the most important effects of the SS/HS Initiative are likely to be observed in changes in what has been termed “school climate.” At the conceptual level, school climate is the perception of the school as an environment in which to grow and learn. For example, a perceived improvement in youth violence and substance use, and hence a safer environment, would be seen as a positive result of the initiative.
Table 1: Evaluation Questions and Data Sources
Evaluation Question |
Data Source |
|
|
|
|
|
|
|
|
2e. Staff School Climate Survey: Purpose and Use of Information
After extensive evaluation, the NET identified the staff version of the California Healthy Kids Survey (CHKS), developed by WestEd for the State of California and available in the public domain, as ideal for this purpose. The CHKS initially was created to meet Federal requirements for the assessment of teacher perceptions of the incidence and prevalence of drug use and violence in the schools. First administered by the state of California in 2004–2005, this staff version of the CHKS was designed to assess seven components of school climate from school staff perspectives:
Student risk/problem behaviors such as substance use, violence, and truancy; the extent of the problem they pose for the school; and the sufficiency of efforts to reduce them
Availability of health and counseling services
Staff and student safety
Nature, communication, and enforcement of school rules and policies
Academic standards and priorities, and learning supports and barriers
Staff-student relationships, school connectedness, and staff supportive relationships
Parent involvement
The CHKS for this data collection consists of 42 items. One item on the survey asks about the respondent’s role at the school, a second asks how long the respondent has been in his/her current position, and a third asks the respondent’s race/ethnicity. The remaining 39 items address the 7 components of school climate listed above. While these components represent the content outline of the instrument, subsequent psychometric analyses have yielded an array of 10 scales and subscales with sufficient reliability to be used in the analysis of school-to-school variation and change over time. These scales, subscales, and obtained reliabilities2 are discussed below (refer to Table 2).
Table 2. California Healthy Kids Survey Reliability
Scale |
No. of Items |
Coefficient Alpha |
Positive Learning and Working Environment |
23 |
0.94 |
Staff/student relationships |
6 |
0.95 |
Student behaviors that facilitate learning |
8 |
0.85 |
School-level norms and standards |
9 |
0.88 |
Staff and Student Safety |
9 |
0.88 |
Perception of student violent behavior |
7 |
0.87 |
Perception of school as safe place |
2 |
0.93 |
Clear, Consistent Communication and Enforcement of Policies |
2 |
0.87 |
Adequate Health/Counseling Services |
3 |
0.70 |
Perception of Problems With Alcohol, Tobacco, and Other Drugs (ATOD) Use |
2 |
0.91 |
3. Use of Information Technology
The Baseline Assessment, Partnership Inventory, School-Level, and Project-Level Survey responses are collected using Web-based methods with a unique login code for the designated respondent. This login process enables the participant in the survey to enter the information directly in an electronically coded format, reducing the burden to both the participants and the administrators of the survey. The login process also enables respondents to complete the survey when it is convenient to them, enhancing the response rate and reducing the perceived burden. Staff School Climate Survey respondents follow a similar procedure, although unique login codes are not assigned to each participant; rather, unique codes are assigned to the respondent’s school. All Web-based applications will comply with the requirements of Section 508 of the Rehabilitation Act.
Historically, the NET has received more than 95 percent of all annual survey responses electronically. However, if completing surveys via the Internet is inconvenient, respondents can complete them in paper form (obtained by email from the NET).
4. Efforts to Identify Duplication
The survey instruments and site visit protocol used to collect data for the evaluation of the SS/HS Initiative are unique. They are tailored to the knowledge of the grant operations of each set of stakeholders and focus solely on information pertinent to planning, implementing, and sustaining the grant activities. The match of instrument to stakeholder is depicted in Table 3. The national evaluation will ask all key stakeholders (i.e., project directors and key partners or Core Management Team members) to complete one quantitative survey instrument and qualitative interview protocol per year. Staff in schools targeted by the grant will be asked to complete one quantitative survey per year. In some rare instances, depending on the random sample for the Staff School Climate Survey, certain individuals may be asked to complete two quantitative surveys, although the estimated burden is less than 1 hour.
Table 3. Expectations for Stakeholder Involvement in Evaluation
Stakeholder Category |
Nature of Process Data |
Instrument |
Local Project Director |
|
|
|
|
|
Representative of Schools Receiving Grant Services |
|
|
Representative of Other Local Grant Partner Organizations |
|
|
The Staff School Climate Survey planned for use in outcome evaluation is an instrument with proven psychometric validity and reliability, as validated by pilot-testing by the State of California and 2 years of administration under the previous national evaluation contract. Certain California schools already complete the instrument biannually to meet data collection requirements for State and Federal educational funding.
Approximately 16 percent of the current grantees are California LEAs that already are required by the State government to conduct the survey biannually. In those years when they already are conducting the survey, depending on response rates and sampling procedures, the California grantees will not be asked to complete the instrument a second time to comply with the needs of the SS/HS Initiative. In effect, the majority of grantees in California will be required only to increase the frequency of survey operations from biannually to annually.
Several other school districts and LEAs that have received or applied for SS/HS Initiative grants also use instruments that are purported to measure school climate. Detailed review of these locally developed instruments found that (1) they lack empirical confirmation of reliability or validity, and/or (2) they are far less comprehensive in their coverage of the multiple dimensions of school climate. On this basis, the NET has determined that similar information is not available from other data collection in place. In fact, several grantees outside of California have informed the U.S. Department of Education they voluntarily intend to replace their locally developed school climate assessments with the Staff School Climate Survey.
5. Involvement of Small Entities
The collection of information applies only to selected individuals in LEAs that have received a specific Federal grant award or that are subgrantees of such school districts. This data collection does not significantly involve small businesses or entities.
6. Consequences if Information Is Collected Less Frequently
The collection of information for the national evaluation of the grant is scheduled to occur on an annual basis during the 4 years of the grant. Annual data collection encourages the accuracy of up-to-date information on recent grant planning and implementation activities and reduces the possibility of confusing change over time resulting from the initiative with short-term change caused by a local event or trend, such as a single, highly publicized act of violence. It also takes into consideration the likelihood of turnover among representatives of key local stakeholders in the grant.
Failure to collect the information on this annual schedule also would prevent the Federal partners from meeting their obligations under GPRA and the Program Assessment Rating Tool (PART) to report to Congress on the outcome and impact of the SS/HS Initiative. Collecting the information only twice (i.e., at baseline and at the end of the grant) would also deprive the grantees of the opportunity to review interim results on aspects of school climate and take corrective action, if necessary, before the end of the grant.
7. Consistency With the Guidelines in 5 Code of Federal Regulations (CFR) 1320.5(d)(2)
The data collection is consistent with the guidelines in 5 CFR 1320.5(d)(2).
8. Consultation Outside the Agency
The 60-day notice required in 5 CFR 1320.8(D) was published in the Federal Register on May 16, 2011 (Volume 76, p.28238). No comments were received.
Consultations on the research design and data sources were conducted during the research design phase and continue to take place as the study design is being finalized. The purpose of these consultations is to ensure the integrity of the study design and the relevance of the data collection activities and to increase the likelihood that the findings of this study will generate valuable information regarding the impact of the SS/HS Initiative on students and communities.
During the design phase where revisions and adjustments to the existing evaluation design and data analysis plan were considered, the NET piloted revised instruments and the new Baseline Assessment Survey with six project directors from previous grantees. After this pilot, the NET convened an Expert Panel Meeting on January 14, 2011, to discuss the proposed approaches for the evaluation. This panel offered valuable recommendations regarding the evaluation design and data collection instruments.
SAMHSA, Department of Education staff, Expert Panel members, and previous grant project directors who have provided guidance on the present study are listed below:
Carmen Arroyo, Special Expert, SAMHSA
Jessica Barnes, External Workgroup Member, Michigan State University
Dave Currey, External Workgroup Member, Pacific Institute for Research and Evaluation
Karen Dorsey, Federal Project Officer, Department of Education
Jim Emshoff, External Workgroup Member, Georgia State University
Rodney Fitzgerald, Former Project Director, Bradley County School District
Dan Flannery, External Workgroup Member, Kent State University
Debby Gaffney, Former Project Director, North Thurston Public Schools
Janet Goodliffe, Former Project Director, Madison School District
Rosemary McCain, Former Project Director, Vail School District #20
Linda Perez, Former Project Director, Pajaro Valley Unified School District
Mike Shain, External Workgroup Member, The AIM Institute
Pam Smith, External Workgroup Member, School District of Lancaster (PA)
Lori Stolee, Former Project Director, Northwest Educational Services District #189
O’Neil Walker, Chief, Mental Health Promotion Branch Division of Prevention, Traumatic Stress and Special Programs Center for Mental Health Services, SAMHSA
Michael Wells, Federal Project Officer, U.S. Department of Education
9. Payment to Respondents
There will be no payments or gifts given or offered to respondents.
10. Assurance of Confidentiality
Protecting the security of data is paramount in any research effort, particularly when sensitive data (information covered by Federal privacy rules, identity of subjects, etc.) are involved. The NET, which has extensive experience conducting national-level evaluations, will continue to follow its existing comprehensive plan to ensure the security of all data collected in the SS/HS national evaluation. All project staff who access any program data will have been trained in data security requirements and security policies (i.e., staff responsibilities for securing hardcopy materials and computer workstations, shredding discarded copies of documents, maintaining security of information collected, etc.). These policies will be shared across the project team and reinforced through training as needed.
For all data collection instruments and data files, there will no individually identifiable information: No personal names will be directly attached to any hardcopy or softcopy form or file. No one except selected project staff will have access to any information that could be used to identify specific individuals. The NET will continue to utilize its Information Resource Management System (IRMS) for collecting and storing qualitative evaluation data. To protect the security of this system, the IRMS has been deployed to a facility maintained by RMC Research Corporation (RMC) that provides monitoring and support, backup heat and cooling, redundant access to the Internet, and space for providing redundant resources to assure high availability. The facility ensures the highest level of security, both physical and virtual. MANILA Consulting Group Inc. (MANILA) and RMC will be responsible for supporting and maintaining the system (hardware and software) through the period of performance. Appropriate safeguards will continue to be used to protect data from improper disclosure in accordance with applicable portions of the U.S. Department of Health and Human Services Automated Information Systems Security Program Handbook and the standards set forth by the National Institute of Standards and Technology. The relevant policies and procedures for each instrument are summarized below.
Security of Site Visit Data. Project staff will continue to assign unique ID numbers to each grantee organization. Each SS/HS site will have a record in the IRMS developed for the SS/HS national evaluation. The record will be the repository of all qualitative information and data for a specific site. Immediately after the initial site visit, site team members will enter the raw data (e.g., notes from interviews or observations and written materials provided by the sites) into the site record. Any names associated with the information collected (other than contact information) will be removed. Identities of individuals supplying information during the site visit will be anonymous in the database.
Security of Partnership Inventory Data. This Web-based inventory is completed by designated representatives of local partnering organizations. The instrument does not ask for the name of the respondent, who is identified only by the organization/agency name and sector (i.e., School District, Law Enforcement, Mental Health Services, or Juvenile Justice).
Security of Project-Level Survey Data. This Web-based survey is completed by the project director for each SS/HS grant site. While the names of the project directors will be known, their individual survey responses will not be shared with anyone outside of NET staff. All data will be reported in aggregate form, except with written permission of the individual project director.
Security of School-Level Survey Data. This Web-based survey is completed anonymously by an individual chosen by the project director to represent each participating school involved with the grant. Thus, the identity of the respondents is not known to anyone other than the project directors, although their general position at the schools will be recorded. Project directors will assist in followup efforts to ensure required completion rates.
Security of the Staff School Climate Survey Data. This survey will be completed by randomly selected staff at each participating school. Rather than generating and tracking an estimated 25,000+ unique respondent codes, the following procedures will be used:
Each project director or a designee will be responsible for obtaining numbered, alphabetized lists by school of eligible staff to use with the sampling plan developed by NET staff.
The project director will select identified staff based off the numbered, alphabetized list and send out survey links to the identified staff.
The project director will be sent weekly reminders on response rates.
The project director or designee will be responsible for ensuring required completion rates, along with encouragement and assistance from NET staff and Federal Project Officers.
Summary. All data collection instruments and data collection procedures have been carefully constructed to avoid any potential issues with data that may raise security concerns. All physical documents containing program data (e.g., faxes, handwritten surveys) will be stored in a secure central location by NET staff charged with their safekeeping. For any sensitive data stored electronically, user IDs and passwords will be required for access. Both the Web server and database will reside in a staffed data center with firewall protection. The database will not allow anonymous connections, and account information is encrypted. If a security incident occurs, proper incident response procedures will be followed. Supervisors are responsible for ensuring all project staff observe all security requirements and receive appropriate security training. Reports and publications from these data are limited to aggregate data analysis that fully protects the identity of individual participants. No data are stored with identifying respondent information.
11. Questions of a Sensitive Nature
Respondents will not be asked any questions of a personally sensitive nature. The subject matter of the interview and survey questions will be limited to the perceptions of grant planning and implementation activities among key stakeholders of the grants and to school employees’ perception of student behavior, substance use, violence, safety, and access to mental health services.
12. Estimates of Annualized Hour Burden
Table 4 provides the basis of the resulting estimates of the hour burden of collection of information, based on field tests of the proposed protocols and instruments.
Table 4. Elements of Annualized Hour-Cost Burden of Data Collection*
Instrument Description |
Anticipated Number of Respondents |
Responses per Respondent |
Average Hours per Response |
Total Annual Hour Burden |
Hourly Rate |
Total Annual Cost |
Site Visit Protocol |
100 |
1 |
9 |
900 |
$32.14 |
$28,926 |
Baseline Assessment Survey |
25 |
1 |
.67 |
17 |
$27.21 |
$456 |
Partnership Inventory |
400 |
1 |
0.25 |
100 |
$32.83 |
$3,283 |
Project-Level Survey |
100 |
1 |
0.42 |
42 |
$27.21 |
$1,143 |
School-Level Survey |
2,300 |
1 |
0.42 |
966 |
$28.26 |
$27,299 |
Staff School Climate Survey |
25,200 |
1 |
0.117 |
2,948 |
$28.26 |
$83,321 |
Total |
28,125 |
|
|
4,973 |
|
$144,428 |
* Number of respondents based on an estimated annual average of 100 grantees. Baseline Assessment Survey administered only to grantees in the 2011–2013 cohorts. School-Level Survey estimates based on an average of 23 schools per grant. Staff School Climate Survey estimates based on 252 respondents per grantee. Average hours per response based on previous evaluation and pilot tests.
13. Estimates of Annualized Cost Burden to Respondents
The National Education Association estimates a nationwide average salary for instructional staff in public schools of $58,780, or slightly less than $29 per hour, for the School-Level Survey and School Climate Survey. Rates for the Project-Level Survey and the Baseline Assessment Survey were based on a nationwide average salary of $56,600, or $27.21 per hour for social and community service managers. The site visits include rates based on average nationwide salaries of common representatives from the required partners, the project director, and the local evaluator. This average annual salary from all five entities is approximately $66,860. The partnership inventory average annual salary ($68,293) is derived from the salaries as the site visit, minus the project director and local evaluator. As a result, our estimate for the total annual cost burden to respondents in the equivalent to their full-time salary is $144,428.3
14. Estimates of Annualized Cost to the Government
The annual cost to the Government of the proposed data collection consists of 40 percent of the Government Project Officer’s time and 15.5 percent of a competitive contract awarded for the conduct of the SS/HS national evaluation to MANILA by the Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. The estimated annual cost of these expenses is $1,435,035 per year.
15. Change in Burden
Currently, there are 16,431 hours in the OMB inventory. The Program is requesting 4,973 hours. The program change in burden is due to a decrease in the number of respondent that are being requested to complete the Staff School Climate Survey (106,675 to 25,200).
16. Time Schedule, Publication and Analysis Plans
16a. Time Schedule
The time schedule for implementing and using the proposed instruments is summarized in Table 5. A 3-year clearance is requested for this project.
Table 5. Time Schedule for SS/HS Evaluation Instruments
Tasks |
Dates |
OMB approval |
Fall 2011 |
Initial data collection |
As soon as OMB approval is received |
Final data collection |
February 2014 |
Data analysis |
Ongoing |
16b. Publication Plans
As noted earlier in the initial discussion of the requirement for this data collection effort, 42 U.S.C. 2099(hh) emphasizes publication and dissemination of the results of information derived from the evaluation of each project funded by the SS/HS Initiative grant program. The evaluation contract for the SS/HS Initiative grant program anticipates that aggregate results from the national evaluation will be incorporated in text and charts of the following publications, planned for completion and distribution in 2011 to 2015:
Quarterly and annual technical reports, including performance and evaluation sections
Quarterly grantee newsletters
Brief reports
A promotional report for dissemination
The three agencies sponsoring the SS/HS Initiative (the U.S. Department of Health and Human Services, the U.S. Department of Education, and the U.S. Department of Justice) may also choose to incorporate aggregate results from collected data in journal articles, scholarly presentations, and congressional testimony referring to the outcomes of the SS/HS grant program.
16c. Analysis Plan
The NET, in collaboration with the Federal Evaluation Workgroup, has examined the question of what is expected to happen in a community as a direct result of the SS/HS Initiative. To explore and explain the potentially complicated relationships, the NET developed a Program Theory Model that identifies the environmental conditions and resources at the time of grant award and shows how the activities associated with the initiative transform these existing conditions into desired near-term and long-term outcomes. The model also looks at factors that may affect both grantees’ ability to break out of any existing organizational silos to better serve youth, and the ability of the resulting collaborative efforts to generate measurable improvements. Key components of the model are described below.
SS/HS grant applicants provide information about existing conditions and resources that are likely to shape the planning of the local project. These conditions and resources include:
Collaborative mechanisms at grant award. SS/HS grant applicants describe a relationship among the public schools and at least two of the following: public sector mental health, law enforcement, and/or juvenile justice. SS/HS is unique in requiring that this relationship predate submission of the application by at least 6 months with a record of previous accomplishments.
Project and community characteristics. Community characteristics describe the grantee’s surrounding community (e.g., population density, education, unemployment).
Pre-grant system resources. Pre-grant system resources include pre-grant investments to address youth needs (e.g., school-based health centers, evidence-based prevention programs already in place) and funding levels and the role of grant funds in the school and community. Some resources may represent assets whose utilization can be improved through collaborative efforts.
Grant capacity. A local grantee’s capacity for the grant includes any existing systems and processes that may aid or hinder its ability to build partnerships and implement programs and activities quickly and effectively.
Comparative analysis of these variables is critical to understanding how conditions and resources in the pre-grant environment facilitate the implementation of SS/HS at both the LEA and local levels.
Structural partnership attributes. The structural attributes of SS/HS partnerships, which are likely to be influenced by characteristics of the pre-grant environment, may impact partnership functioning and near- and long-term outcomes. Two key attributes are partnership composition (partner organizations are actively involved in the partnership) and organization (the structure of the partnership such as developing committees), both of which may remain consistent or change over the duration of the grant based on local needs and challenges. The NET will examine whether partner organizations involved in the SS/HS project have changed over time. While the NET has not yet found a predictive relationship between the addition of partner agencies and improved outcomes, the NET anticipates exploring the proportion of community agency involvement and its relationship to improved near-term outcomes.
Partnership organization refers to the manner in which a partnership structures itself to assess community needs; identify, implement, and monitor best practice solutions; and plan for sustainability. For example, some grantees may have one single collaborative group throughout the life of the grant, while other grantees have a core executive management group, a broader advisory group, and other committees. The NET will examine whether there is a typical organizational configuration among SS/HS sites within and across cohorts and whether there are common patterns of change in organizational structure over time.
Functional partnership attributes. Partnership functioning refers to how the collaborative members work together on grant activities as a whole. The NET assumes that the manner in which they work together will impact the effectiveness of program implementation and sustainment of system changes. The SS/HS partnerships often begin functioning prior to the grant award as part of the means to prepare for the grant application. The nature of collaborative functioning is likely to change over time as relationships develop, goals are accomplished, or challenges are encountered.
The NET will examine five elements of functional partnership attributes that may be important determinants of achieving desired system changes and long-term behavior changes:
Interaction among partners. Collaborative members who interact more frequently and extensively in planning and implementing major components of the project may be more effective. Partners that have minimal interactions may experience less communication about project activities and feel “out of the loop.”
Decisionmaking. Research on collaborative functioning supports the idea that involving partners in decisionmaking is important in achieving the project’s goals. Understanding the formality of operational procedures, communication practices, and breadth of decisionmaking may contribute to explaining the relationship, and better understanding the role of partnership organization generally. Butterfoss (2007)4 cites a study that found that the more routinized procedures become in an organization, the more likely they are to be sustained. Decisionmaking will focus specifically on partner organizations’ influence on coalition policies and actions.
Level of partner contributions. The NET will examine the role each partner has played in the overall project and in five types of partnership activities: planning, implementation, monitoring/tracking performance, sustainability planning, and formulating policy change. Given the emphasis on multiagency activities and service integration and coordination among organizations, the NET assumes that the most effective partnerships will have significant contributions from a broad range of partners across a broad range of activities.
Overall partnership functioning. The NET will also examine partner perceptions of how well the collaborative is functioning in its community. Observed variation will enable the NET to address how different patterns of collaborative functioning relate to system changes and long-term behavior changes. In particular, it may be important to distinguish among the effects on outcomes when community partners engage in collaborative functioning before grant award, after grant award, and both before and after the grant award. The findings may provide useful information on both the timing and the activities of collaborative functioning that are related to systems change.
Leadership. Shared leadership and bridge-building have been tied to effective leadership (Alexander, Zakocs, Earp, & French, 2006).5 Shared leadership involves fostering the active involvement of others in leadership beyond participation in decisionmaking. The NET will focus on the role leadership plays in the partnership.
Operating Environment. Near- and long-term outcomes also may be influenced by factors external to the partnership that constitute the operating environment. For example, barriers to implementation such as staff turnover or transportation issues in a geographically isolated area can significantly affect a site’s progress. Conversely, a site may experience facilitators to implementation such as partners with a strong commitment to the project or data to support the effectiveness of programs. The breadth and depth of barriers and facilitators within a site can be a predictor of a site’s success in implementing the project and achieving its goals.
In some jurisdictions, one or more individuals may serve as community advocates for the SS/HS project. Advocates can provide inspiration and commitment to use the grant to sustain integrated youth development outcomes. Effective advocacy is believed to help the SS/HS partnership maintain a coherent body of policy and program interventions designed to reinforce positive effects on youth behavior and the school environment. The NET will explore this concept in its site visits to see if certain types of individuals, such as superintendents, influence project outcomes.
While the size of the SS/HS grant is substantial in many sites, some sites lack basic resources needed to implement the project as intended. Obtaining school-level perceptions of the importance of the SS/HS resources, in addition to partner perspectives on resource availability, is critical in putting the long-term outcomes into context. For example, a site that lacks needed resources may not experience improvements in grant goals regardless of what partnership activities are undertaken.
Grantees are required to conduct a local evaluation of the grant. Discussions with the local evaluation team can reveal factors such as the extent of local input, problems with data collection and other barriers, and community involvement in the dissemination of data that can have an impact on near- and long-term outcomes. Problems or nuances with the local evaluation data are important to understand as these issues affect the usability of the GPRA data as long-term outcomes. A review of final evaluation reports will likely yield relevant information on this topic.
Many, if not most, of the SS/HS activities and programs are implemented in individual schools rather than in the community. The success of these initiatives depends to a large degree on the involvement of the schools in the planning and implementation decisions that affect them. The NET anticipates this involvement might be demonstrated in several ways:
Involvement of school representatives in SS/HS planning and decisionmaking
Increased collaboration with SS/HS core partners
Ultimately, positive outcomes with regard to violence, substance abuse, mental health, and school climate
The near-term results of the SS/HS Initiative grant activities consist of three closely related changes believed to be related to improved school climate, sustainability, and other long-term outcomes: (1) comprehensive policies and practices, (2) implementation of enhanced services, and (3) improved coordination and service integration among the partners.
Comprehensive policies and practices. Programs and activities implemented or enhanced as a result of the local SS/HS Initiative are expected to be comprehensive in scope (the topics they address). The NET plans to use the number of new or expanded activities, as indicated in the initial Baseline Assessment and annual Project-Level Survey, as a proxy for implementation of policies and practices. For example, instituting an evidence-based parenting curriculum may yield significant improvements in school readiness among preschool youth. However, a community that institutes training for child care providers, policies to screen for preschool youth at risk, and referral procedures—in addition to the parenting curriculum—is making a larger investment in time and effort to early childhood development than a community that institutes the parenting curriculum alone.
Collection of survey data measuring diffusion of implementation to the school level provides the NET with an opportunity to explore whether long-term outcomes in a given program element are affected by the diffusion of programs and services to individual schools. The primary source of this information is the School-Level Survey, which asks a designated staff person in each targeted school to report on the changes he or she observes occurring within the school.
Implementation of enhanced services. Another aim of the SS/HS Initiative is to enhance services. This is evidenced by the SS/HS grant’s emphasis on implementation of services, programs, and activities that have been selected by grant staff and/or the local partnership as “best” (i.e., most appropriate for the community needs). Where possible, it is assumed that best practices include evidence-based programs and curricula. For the national evaluation, the issue is determining not only whether services are enhanced by SS/HS grant activities in the near term, but also whether these enhanced services translate into the desired long-term outcomes.
The NET plans to use the numbers and types of new or expanded evidence-based programs implemented in the Project Level Survey as a proxy for the extent of service enhancement for each program area. Qualitative information gathered from site visits will improve the NET’s understanding of service enhancement. As there is some subjectivity involved in these types of self-reported assessments, the NET uses site visits as opportunities to validate and cross-reference responses. Specifically, probes help determine whether service innovations are being actively monitored in some objective way. This can help provide some assurance that the local assessments are based (at least in part) on objective data.
Improved coordination and service integration among schools and partner agencies. This refers to the hypothesis that agencies in a grantee community’s system of youth development services will collaborate more in planning, implementing, monitoring, and sustaining activities. Improved coordination among the agencies responsible for addressing the problems of the community’s youth is another type of system change that constitutes a potential long-term outcome. Though difficult to measure directly, improved coordination may affect the long-term outcomes according to changes in youth perceptions and behavior and school climate.
The NET will use the Baseline Assessment, Project-Level Survey, and the Partnership Inventory to assess whether interagency coordination has improved within the community and monitor several promising indicators of collaboration. The results of this evaluation will help the Federal partners assess whether enhanced collaboration is generated by the SS/HS grant requirements. Ultimately, this will assist in determining whether a culture of collaboration is associated with improvements in measures of violence, substance use, and access to mental health services among youth, in overall school climate, and in perceived sustainability of effort following the end of the grant period.
17. Display of Expiration Date
The expiration date will be displayed.
18. Exceptions to Certification Statement
There are no exceptions to the certification statement. The certifications are included in this submission.
1 The Federal Evaluation Workgroup is composed of officials representing two agencies: (1) the U.S. Department of Education, Office of Safe and Drug-Free Schools; and (2) the U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.
2 Cronbach alpha reliabilities computed for each scale based on the 2009–2010 administrations of the survey (N = 29,192–33,432, depending on the scale)
3 These salary estimates were calculated from data obtained from the Department of Labor’s Occupational Information Network (O*NET) Web site http://online.onetcenter.org on January 10, 2011, and are based on 2009 median wages.
4 Butterfoss, F. D. (2007). Coalitions and partnerships in community health. San Francisco: Jossey-Bass.
5 Alexander, M., Zakocs, R., Earp, J., & French, E. (2006). Community coalition project directors: What makes them effective leaders? Journal of Public Health Management Practice, 12(2), 201–209.
| File Type | application/msword |
| File Title | The Safe Schools/Healthy Students (SS/HS) National Evaluation |
| Author | Gary Hill |
| Last Modified By | DHHS |
| File Modified | 2011-09-20 |
| File Created | 2011-09-20 |