SPF-SIG-OMB-SS-08-12-09Final

Part A. Justification

A1. Circumstances Necessitating Data Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP) requests OMB approval for revision to the protocol for the ongoing cross-site evaluation for the Strategic Prevention Framework State Incentive Grant (SPF SIG) (OMB No. 0930-0279) which expires on 09/30/09. This revision includes two parts:

1. Continuation of the use of the previously approved two-part Community Level Instrument (CLI Parts I and II) for Cohorts I and II.

2. The use of three additional instruments to support the SPF SIG Cohorts III and IV Cross-site Evaluation. All three instruments are modified versions of data collection protocols used by Cohorts I and II. The three instruments are:

   1. a Grantee-Level SPF Implementation Instrument,

   2. a Grantee-Level Infrastructure Instrument, and

   3. a two-part Community-Level SPF Implementation Instrument.

CSAP is funding two cross-site evaluations of the Strategic Prevention Framework State Incentive Grant (SPF SIG), one focusing on Cohorts I and II and the other focusing on Cohorts III and IV. Collectively, these evaluations of the SPF SIG program provide an important opportunity for the field of prevention.

Every attempt has been made to make the evaluation for Cohorts III and IV comparable to Cohorts I and II. However, resource constraints for the Cohorts III and IV evaluation have necessitated some streamlining of the original evaluation design. In addition, because the ultimate goal is to fund all eligible jurisdictions, there are no control groups at the grantee level for Cohorts III and IV. The primary evaluation objective is to determine the impact of SPF SIG on the reduction of substance abuse related problems, on building state prevention capacity and infrastructure, and preventing the onset and reducing the progression of substance abuse, as measured by the SAMHSA National Outcomes Measures (NOMs). Data collected at the grantee, community, and participant levels will provide information about process and system outcomes at the grantee and community levels as well as context for analyzing population-level and participant-level outcome measures. The Community-Level Part I and Part II Instrument used by Cohorts I and II and the Grantee-Level Infrastructure and SPF Implementation Instruments and the two-part Community-Level SPF Implementation Instrument used by Cohorts III and IV are included in this OMB review package and are the main focus of this request.

A1a. The SPF SIG Program

The SPF SIG project is authorized under Section 501 (d)(4) of the Public Health Service Act (42 USC 290aa). The SPF SIG is a major SAMHSA Infrastructure Grant program that supports an array of activities to help states and communities build a solid foundation for delivering and sustaining effective substance abuse prevention services. The SPF SIG is implemented by CSAP and is designed to: (1) prevent the onset and reduce the progression of substance abuse, including childhood and underage drinking; (2) reduce substance abuse-related problems in communities; and (3) build prevention capacity and infrastructure at the state/territory and community levels. CSAP provides funding to states, Pacific jurisdictions, and tribal entities to implement the five steps of the strategic prevention framework (SPF), which are:

Step 1: Profile population needs, resources, and readiness to address the problems and gaps in service delivery;

Step 2: Mobilize and/or build capacity to address needs;

Step 3: Develop a comprehensive strategic plan;

Step 4: Implement evidence-based prevention programs, policies, practices and infrastructure development activities; and

Step 5: Monitor process, evaluate effectiveness, sustain effective programs/activities, and improve or replace those that fail.

In FY 2004, CSAP funded Cohort I, which consisted of 21 states and territories, for up to five years to implement the SPF SIG program. Cohort II was funded in FY 2005 and includes five additional states and territories. Cohort III, which was funded by CSAP in FY 2006, includes 10 states, 1 Pacific jurisdiction, and five tribal entities. An estimated 20 Cohort IV sites will be funded in FY 2009. For the purposes of this document the word grantee will refer to all funded states, Pacific jurisdictions, and tribal territories.

A1b. The Cross-site Evaluations

CSAP has funded two cross-site evaluations of SPF SIG, one focused on Cohorts I and II and the other focused on Cohorts III and IV. The SPF SIG is the first broad-based, data-driven effort that simultaneously attempts to influence both strategic planning and prevention systems at the jurisdiction and community levels, as well as implement evidence-based prevention interventions in communities. These evaluations will help determine whether the SPF SIG has met these expectations and, if so, under what conditions.

A1b1. Cohorts I and II Cross-site Evaluation

The National Institute on Drug Abuse (NIDA) is providing support to CSAP to evaluate the impact of the SPF SIG project for Cohorts I and II. Since funding for the evaluation began in September 2004 and OMB clearance was received in 2006, this evaluation is already in process. Information on the overall evaluation is presented below as context for understanding SAMHSA’s request to revise the timeline on the previously approved CLI instrument through 9/30/12 so that additional rounds of CLI data can be collected.

The Cohorts I and II cross-site evaluation team is currently implementing a multi-method quasi-experimental evaluation of the SPF SIG project at national, state, and community levels. A major objective of the SPF SIG evaluation is to determine the impact of SPF SIG on the SAMHSA National Outcome Measures (NOMs), and to assess the impact of the program as a whole. The data from the CLI (Parts I and II) will be used to interpret the impact of the SPF SIG on all of the NOMs domains related to prevention (i.e., Abstinence, Education/Employment, Crime and Criminal Justice, Access/Capacity, Retention, Cost Efficiency and Use of Evidence-based Practices). The evaluation is also measuring: the effect of establishing and sustaining infrastructure at the state and community-levels to allow for data-based decision-making; the implementation of the SPF; and environmental factors that affect substance abuse.

Both quantitative and qualitative data are being gathered as a part of the SPF SIG cross-site evaluation for Cohorts I and II. Specifically, data are being collected from the 26 states and territories receiving grants in 2004 (Cohort I sites) and 2005 (Cohort II sites) and as many as 32 non-Cohort I and II grantee states and territories that will serve as a comparison group. Data sources include: (1) grantee Epidemiology and Outcome Workgroups (EOW) and communities, (2) state-level evaluations, (3) existing national- and state-level population-based indicators, (4) standardized data collected by the evaluators on the implementation of the SPF, and (5) archival sources such as grant applications and State Prevention Advancement and Support Program (SPAS) reports.

The timing of the Cohort I and II evaluation, which began concurrently with the funding of the program, allowed the team to gather meaningful baseline data. However, since the majority of the Cohort I grantees are expected to receive one year no cost extensions and the Cohort II grantees will not complete their original period of performance until FY 2010, a continuation of CLI data collection through FY 2012 will be required to allow the team to observe community-level accomplishments within SPF SIG states throughout the full life-cycle of the program.

A1b2. Cohorts III and IV Cross-site Evaluation

The Cohort III and IV cross-site evaluation team is implementing a multi-level evaluation design encompassing data collection at the grantee, community and participant levels. Data will be gathered from the 16 states, Pacific jurisdictions, and tribal territories receiving grants in FY 2006 and an estimated 20 Cohort IV grantees that may be funded in FY 2009.

In accordance with CSAP’s program goals to assess the impact of the SPF on measurable quantifiable outcomes, a major focus of the evaluation is on impact. However, the collection of process data at the grantee and community level is necessary for describing and documenting the activities undertaken as part of the SPF SIGs and supporting the results of project outcomes.

The evaluation design will rely heavily on the use of standardized self-report data collection instruments specifically designed to collect information on known mediators, moderators, and outcomes of interest to the evaluation plan. The data from the two revised grantee-level instruments and the revised community-level instrument will specifically measure: the effect of establishing and sustaining infrastructure at the grantee and community-levels to allow for data-based decision-making; the implementation of the Strategic Prevention Framework; and environmental factors that affect substance abuse. Recognizing that all grantees have prevention activities already underway, the collection of baseline data using these instruments will account for pre-SPF SIG activities in estimating the effects of SPF SIG-initiated activities. In addition, these data will be used to assess whether the steps of the framework were fully implemented as intended, thereby avoiding the attribution that any lack of effect is due to the SPF itself rather than failure to implement steps of the framework or implementing them improperly or incompletely. The process components of the SPF SIG evaluation will allow the evaluators to disentangle the effects of various project-related activities and help identify which program and policy elements are effective, under what conditions, and with which target populations.

A2. Purpose and Use of Information

The SPF SIG is a major investment by the Federal Government to improve state substance abuse prevention systems, and enhance the quality of prevention programs, primarily through the implementation of the SPF. The goal of this initiative is to provide states, Pacific jurisdictions, tribal entities and communities within them with the tools necessary to develop an effective prevention system with attention to the processes, directions, goals, expectations, and accountabilities necessary for functionality. SAMHSA/CSAP needs to collect information over the course of the grant period to monitor the progress of the SPF SIG initiative, particularly the implementation of evidence-based practices by communities. CSAP will use the findings from the cross-site evaluations to assess the implementation of the SPF, infrastructure development at the grantee and community level, and the outcomes achieved by this initiative. Without these data the impact of the SPF SIG will be unknown. Additionally, findings from these evaluations may assist CSAP policymakers and program developers as they design and implement future initiatives.

A2a. Cohorts I and II Cross-site Evaluation

The primary sources of data for the Cohorts I and II cross-site evaluation consist of instruments implemented at the grantee (state) and community-levels.

A2a1. Grantee-Level Instruments

Data collection using the State Implementation and State Infrastructure Interview protocols will be completed within the current period of OMB approval for Cohorts I and II.

A2a2. Community-Level Instrument (CLI)

The CLI is a two part, web-based survey for capturing information about SPF SIG implementation at the community level (originally submitted as an addendum to OMB No. 0930-0279). Part I (Attachment A1a) of this instrument was developed to assess the progress of communities as they implement the Strategic Prevention Framework (SPF), and Part II (Attachment A1b) was developed to gather descriptive information about the specific interventions being implemented at the community level and the populations being served including the gender, age, race, ethnicity, and number of individuals in target populations. Each SPF SIG funded community will complete a separate Part II form for each intervention they implement.

The Community-level Instrument (Parts I and II) was designed to be administered two times a year (every six months) over the course of the SPF SIG initiative. The Cohorts I and II cross-site evaluation team plans to collect data for two more years once this request for a revision is approved. Data from this instrument will allow CSAP to assess the progress of the communities in their implementation of both the SPF and prevention-related interventions funded under the initiative. The data may also be used to assess obstacles to the implementation of the SPF and prevention-related interventions and facilitate mid-course corrections for communities experiencing implementation difficulties.

In keeping with the objectives of the Cohorts I and II cross-site evaluation, data from the CLI (Parts I and II) will also be used to assess the relationship between SPF implementation and changes in the NOMs. Additionally, data from this instrument will be used to assess the types of interventions being implemented in communities that receive SPF funds and changes in prevention infrastructure at the community level. Prevention infrastructure refers to the organizational characteristics of the system that delivers prevention services, including all procedures related to planning, data management systems, workforce development, intervention implementation, evaluation and monitoring, financial management, and sustainability. All of the data from this instrument will be used to determine what accounts for any variation in the NOMs. Without these data, it would be impossible to determine how the SPF SIG initiative had an impact on changes in the NOMs or which components of the SPF process were responsible for the observed changes.

A2b. Cohorts III and IV Cross-site Evaluation

The primary sources of data for the Cohorts III and IV cross-site evaluation will include instruments implemented at the grantee- community- and participant-level.

A2b1. Grantee-Level Instruments (GLI)

Two web-based surveys, GLI Infrastructure Instrument and GLI Implementation Instrument, were developed for assessing grantee-level efforts and progress (Attachments B1a and B1b). Both instruments are modified versions of the face-to-face interviews used in the SPF SIG Cohort I and II Cross-Site Evaluation (OMB No. 0930-­0279).

The original Cohort I and II interview protocols were developed to assess the implementation of the SPF process at the grantee-level and measure the development of jurisdiction-wide systems to manage prevention services. The interview protocols were developed by the Cohorts I and II cross-site evaluators using an iterative approach combining findings from the empirical literature, CSAP documents, lessons learned from the State Infrastructure Grant (SIG) program, and input from SPF-SIG grantee stakeholders solicited during interviews and via feedback on drafts. The Infrastructure protocol was used to assess Alcohol, Tobacco and other Drug (ATOD) prevention capacity within various domains at the grantee system level. It captured infrastructure development activities that occurred as a consequence of SPF but also those that resulted from other causes. The SPF Implementation protocol was more normative in character and directly assessed each grantee’s implementation of the 5 SPF steps and was limited to actions that have occurred as a direct result of the SPF including the implementation of the EOW. Both protocols were implemented via a telephone survey of key informants in each jurisdiction, conducted annually.

Because of resource constraints, the Cohorts III and IV evaluation team modified the original instruments to eliminate evaluation staff involvement in the data collection effort and reduce the amount of time required to contextualize narrative data into categorized data. Every attempt was made to preserve the content areas to allow for the collection of comparable information across all 4 cohorts. The original interview protocols were adapted into a survey format by replacing the majority of open-ended questions with forced-choice-response questions using data collected from the Cohort I and II grantees to indentify common themes and using the existing descriptive anchors developed by the Cohort I and II evaluation team when available. The original protocol was also shortened by eliminating questions that did not produce discriminating information.

Both the Grantee Infrastructure Instrument and the Grantee SPF Implementation Instrument will be completed by the grantees’ evaluators twice over the life of the SPF SIG award. The grantee’s evaluator is strongly encouraged to obtain input from others involved with the SPF SIG funded project. As part of this process, we encourage the local evaluator to complete and review responses with key individuals, such as the project coordinator, members of the EOW and SPF SIG Advisory Council, prevention agency staff, and others, as appropriate. Detailed administration Question by Question Guides have been developed to help improve the reliability and validity of the data collected thereby ensuring quality data with which to evaluate grantee-level progress.

GLI Infrastructure Instrument

The Infrastructure Instrument collects information with regard to the operations of the overall prevention system in the jurisdiction (i.e., the entire set of agencies, organizations, and persons that contribute to efforts to prevent substance abuse and related problems within the jurisdiction), not just the SPF SIG project. The GLI Infrastructure Instrument is designed to collect information about a specific snapshot in time. The purpose of the baseline data collection of the GLI Infrastructure Instrument is to gather information about how the overall prevention system was structured and functioned at the time the grant was awarded. A second collection of the GLI Infrastructure Instrument (follow-up data collection) will occur near the completion of the grant. The purpose of the follow-up data collection of the GLI Infrastructure Instrument is to gather information about how the overall prevention system was structured and functioned 5 years after the grant was awarded.

GLI Implementation Instrument

The GLI Implementation Instrument collects information specific to the execution of the 5 steps of the Strategic Prevention Framework in the jurisdiction. Data collected from the instrument will be used to evaluate the effectiveness of the Strategic Prevention Framework. Baseline data collection is designed to provide a retrospective picture of the period of time during the development and approval process of the strategic plan and is expected to be completed near the date of the approval of the strategic plan. A second collection of the GLI Implementation Instrument (follow-up data collection) will occur approximately 36 months after the approval of the strategic plan. The purpose of the follow-up data collection of the GLI Implementation Instrument is to gather information about ongoing activities related to the SPF planning steps.

A2b2. Community-Level Instrument (CLI)

The Community-Level Instrument is a two part, web-based instrument for capturing information about SPF SIG implementation at the community subrecipient level (communities that receive SPF SIG funds from the Cohort III and IV grantees) (Attachments B1c and B1d). The instrument is a modified version of the one in use in the SPF SIG Cohorts I and II Cross-Site Evaluation (OMB No. 0930-0279). Slight modifications were made to clarify question intent or refine response items to help improve data quality. Content areas were preserved to allow for the collection of comparable information across all 4 cohorts. Overall reductions in burden were accomplished by reorganizing the format of the original instrument, optimizing the use of skip patterns, and replacing the majority of open-ended questions with multiple-choice-response questions.

Part 1 of the instrument will gather information on the communities’ progress implementing the five SPF SIG steps and efforts taken to ensure cultural competency throughout the SPF SIG process. Subrecipient communities receiving SPF SIG awards will be required to complete Part I of the instrument annually. Part 2 will capture data on the specific prevention intervention(s) implemented at the community level. A single prevention intervention may be comprised of a single strategy or a set of multiple strategies. A Part II instrument will be completed for each prevention intervention strategy implemented during the specified reporting period. Specific questions will be tailored to match the type of prevention intervention strategy implemented (e.g., Prevention Education, Community-based Processes, and Environmental). Information collected on each strategy will include date of implementation, numbers of groups and participants served, frequency of activities, and gender, age, race, and ethnicity of the population served/affected. Subrecipient communities’ partners receiving SPF SIG awards will be required to update Part II of the instrument a minimum of every six months.

Data from this instrument will allow CSAP to assess the progress of the communities in their implementation of both the SPF and prevention-related interventions funded under the initiative. The data may also be used to assess obstacles to the implementation of the SPF and prevention-related interventions and facilitate mid-course corrections for communities experiencing implementation difficulties. Without these data, it would be impossible to determine how the SPF SIG initiative had an effect on changes in community- and participant-level NOMs or which components of the SPF process were responsible for the observed changes.

A2b3. Participant-Level Instrument

Participant-level change will be measured using CSAP NOMs Adult (participants aged 18 or older) and Youth (participants aged 12-17) Programs Instrument. These instrument forms are already approved by OMB (OMB No. 0930-0230). Subrecipient communities will have the opportunity to select relevant measures from the CSAP NOMs Adult and Youth Programs Instrument Forms based on site-specific targeted program outcomes and may voluntarily select additional outcome measures that are relevant to their own initiatives. Participant-level data will be collected from all participants in direct-service programs lasting 30 days or longer. The participant-level instruments will be administered to each participant at program entry, program exit, and six months after program exit to examine the effect of direct service evidence-based strategies on participant-level NOMS outcomes. Cohort III and IV SPF SIG grantees have been included in the currently OMB approved umbrella NOMs application (OMB No. 0930-0230) covering the collection of participant-level NOMs by all SAMHSA/CSAP grantees. Therefore no additional burden for this evaluation activity is being imposed and clearance to conduct the activities is not being requested.

A3. Use of Information Technology

Both of the cross-site evaluations for Cohorts I and II and Cohorts III and IV use information technology to minimize respondent burden.

A3a. Cohorts I and II Cross-site Evaluation

The CLI is a web-based survey and both Part I and Part II will continue to be completed online. The Cohorts I and II cross-site evaluation team has found that web-based administration of this instrument increases the efficiency of data submission and improves data quality. Additionally, completion of this instrument online reduces the burden on communities as some items are pre-filled based on information from the initial submission, and some items in Part II are pre-filled with information from Part I of the instrument.

Technology is also being used to facilitate communication and provide updates to SPF SIG personnel for Cohorts I and II. Through the SPF SIG web board, State evaluators, project directors, coordinators and other key staff have the opportunity to exchange valuable advice and receive announcements and clarifications from CSAP, other SPF SIG States, and the cross-site evaluation team. In addition to the web board, the cross-site evaluation team also sends electronic copies of the guidance and resource materials via e-mail and CD to SPF SIG States upon request. Data from the CLI (Parts I and II) are made available to State grantees and funded communities via the web for online analysis or by downloading for offline analysis.

A3b. Cohorts III and IV Cross-site Evaluation

The Grantee-Level Instruments (the SPF Implementation Instrument and the Infrastructure Instrument) and the Community-Level Instrument (Parts I and II) are web-based surveys and will be completed online through SAMHSA/CSAP’s Services Accountability and Monitoring System (CSAMS). Web-based administration of the instruments will increase the efficiency of data submission and improve data quality. Additionally, completion of the instruments online will reduce the burden on grantees and communities, as some items will be pre-filled based on information from the initial submission. In the Community-Level Instrument, some items in Part II will be pre-filled with information from Part I of the instrument.

The Participant-Level Instrument will also be submitted via a web-based data entry tool. The data entry tool will also be available through SAMHSA/CSAP’s CSAMS. The NOMS-based data entry tools in CSAMS are designed to reflect the structure of the questionnaires. The system allows for the entry of data from completed questionnaires directly into the system. Grantees preferring to create their own data files have the option of uploading complete data files to CSAMS.

CSAMS is maintained by CSAP’s Data Information Technology Infrastructure Center (DITIC). Data entered online by grantees are periodically extracted by DITIC and transmitted in encrypted form to CSAP’s Data Analysis Coordination and Consolidation Center (DACCC). Grantees have two options for accessing the data they enter online. In the first option, grantees can download, in spreadsheet form, the raw data they have entered online as soon as it is submitted. Grantees can also access their data from the cleaned analysis files prepared by DACCC through the Cohort III and IV SPF SIG web board.

Finally, technology will be used to facilitate communication and provide updates to SPF SIG personnel. Through a SPF SIG web board, grantee evaluators, project directors, coordinators and other key staff will have the opportunity to exchange valuable advice; find guidance and resource materials; and receive announcements and clarifications from CSAP, other SPF SIG grantees, and the cross-site evaluation team. In addition to the web board, the cross-site evaluation team will also send electronic copies of guidance and resource materials via email and CD to SPF SIG grantees upon request.

A4. Efforts to Identify Duplication

The information being collected by the cross-site evaluations for SPF SIG Cohorts I and II and Cohorts III and IV is specific to the program and is not available elsewhere.

A5. Involvement of Small Entities

The primary entities for the Cohort I and II and the Cohort III and IV studies are states, jurisdictions, and tribal territories and the communities funded within these entities. Community is broadly defined as the politically or geographically defined area or culturally or epidemiologically defined target population that the grantee chooses for any given prevention intervention. Because grantees and funded communities involve government or tribal agencies, universities, hospitals, or other large organizations, the evaluation will have no significant economic impact on small entities or small businesses.

A6. Consequences If Information Collected Less Frequently

The cross-site evaluation of the SPF SIG program provides an important opportunity for the field of prevention. Not conducting this data collection would significantly impede SAMHSA’s ability to assess the implementation of the SPF SIG process and measure improvements in: strategic planning; capacity and infrastructure development; data-driven decision making; and implementation of evidence-based prevention programs. Less frequent data collection would also impede CSAP’s ability to track changes in substance use and substance use related problems.

A6a. Cohorts I and II Cross-site Evaluation

The CLI instrument is currently being administered twice per year to each State and community that receives SPF funding over the course of three years. A continuation will ensure that comparable data are collected for the remainder of the SPF SIG program for Cohorts I and II.

Experience from the State Incentive Grant (SIG) project, the precursor to the SPF SIG program, as well as discussions with state-level evaluators, indicate that it is necessary to gather this information at least twice per year. Community-level activities change frequently within a year, and staff turnover at the community-level is common. Thus, to ensure the collection of valid and reliable data, data collection needs to occur twice per year. In addition, data from multiple time periods within a year is essential for monitoring the progress of states and communities as they implement the SPF, and for identifying communities that are experiencing obstacles to implementing the SPF. Without data from multiple time periods during the program, it will be impossible to determine whether implementation progress is related to changes in NOMs.

A6b. Cohorts III and IV Cross-site Evaluation

Information will be gathered retrospectively and prospectively to allow the team to gather meaningful baseline data and observation of the SPF SIG grantees and funded communities throughout the life-cycle of the program. The specific schedule of data collection is dependent upon the data being collected: The Grantee-Level Infrastructure and Implementation Instruments are completed twice over the grant period; the Community-Level Instrument (Part 1) is collected annually, the Community-Level Instrument (Part 2) is collected a minimum of every 6-months; and the Participant-Level Instrument will be collected as necessary dependent on how often participant level programs are run. Data from multiple time periods are essential for monitoring the progress of states and communities as they implement the SPF and deliver evidence-based strategies and for identifying communities that are experiencing obstacles and may need technical assistance.

A7. Consistency With Guidelines in 5 CFR 1320.5(d) (2)

The proposed data collection for the Cohorts I and II and Cohorts III and IV cross-site evaluations fully complies with all guidelines of 5 CFR 1320.5.

A8. Consultation Outside the Agency

A8a. Federal Registry Announcement

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on Friday August 22, 2008 (Volume 73, Number 164, pages 49691-49693). No comments were received.

A8b. Consultations Outside the Agency

A8b1. Cohorts I and II Cross-site Evaluation

The Cohorts I and II cross-site evaluation design, data analysis plan, and CLI (Parts I and II) received several rounds of review prior to the original OMB submission. These reviews were the result of ongoing collaboration with two SPF SIG advisory groups, State level evaluators, and program directors.

Consultation with Internal and External Advisory Groups. Members of the SPF SIG External Technical Advisory Group (ETAG) reviewed the cross-site evaluation design, analysis plan, and CLI (Parts I and II).  The ETAG includes a group of SPF SIG project directors and evaluators; evaluation and prevention experts; a representative from the National Institute on Drug Abuse (NIDA); and three SAMHSA staff not directly involved in the evaluation.  Each ETAG member was carefully selected to ensure representation from the following:  Federal and State government staff; local providers; representatives of the national prevention network system (CADCA); and members versed in specialized areas such as cultural competence, environmental strategies, fidelity and adaptation, evaluation design, and data analysis.  Their feedback was incorporated into working and final drafts of the evaluation design, data analysis plan, and CLI (Parts I and II).  These reviewers’ names, titles, organizational affiliations, and current telephone numbers are provided in Attachment A2.

The Cohorts I and II cross-site evaluation team also seeks regular consultation with the SPF SIG Internal Workgroup.  This group meets on a monthly basis at CSAP and consists primarily of CSAP and NIDA staff but also includes two SAMHSA staff outside of CSAP.  As with the External Technical Advisory Group, the Internal Work Group provided feedback on the evaluation design and data analysis plan which was incorporated in working and final drafts.  These reviewers’ names, titles, organizational affiliations, and current telephone numbers are also provided in Attachment A2.

Consultation with Respondents. The SPF SIG Cohorts I and II cross-site evaluation team was responsible for the development and pilot testing of the CLI (Parts I and II).  This team frequently sought consultation with respondents in the development and refinement of this Instrument, as well as the pilot testing of this Instrument. 

In the development of the CLI (Parts I and II), key prevention stakeholders, including State SPF SIG project directors and evaluators and other key SPF SIG staff, were consulted.  They provided feedback on the content and format of the instrument’s domains, indicators, and measures to ensure that they had face validity and were not too burdensome for respondents to answer.  In addition, all SPF SIG States were given the opportunity to review the instrument and provide comments and questions on their content and format.  

The CLI (Parts I and II) was pilot tested in four States in January 2006. The individuals that participated in the pilot test represented the following types of organizations: mental health services, juvenile justice program services, substance abuse prevention services, youth-focused community organizations, and coalitions. Minor changes were made to the instrument as a result of the pilot testing; these are discussed in section B4a. Participants in the pilot test were also consulted on their estimate of the amount of time required to complete this instrument, and the burden associated with this instrument; these are discussed in section A12a.

A8b2. Cohorts III and IV Cross-site Evaluation

This submission is drawn from the one originally submitted by SAMHSA for the cross-site evaluation of SPF-SIG Cohort I and II grantees. The current evaluation design, data analysis plan, and revised instruments received several rounds of review. These reviews were the result of ongoing collaboration with the CSAP SPF SIG project officer, the SAMHSA/CSAP Data Analysis Coordination and Consolidation Center (DACCC) project officer and team members, the DACCC External Steering Committee (ESC), and a grantee-level workgroup consisting of evaluators and project directors. All SPF SIG grantees were given the opportunity to review instruments and provide comments and questions on their content and format. The purpose of such consultations is to ensure the technical soundness of the evaluation, and to verify the importance, relevance, and accessibility of the information sought in the evaluation and to insure that this type of monitoring will continue to take place throughout the evaluation.

Members of the DACCC, the ESC, and other outside expert participants including researchers, evaluators, state representatives, and grantees who participated in these processes may be found in Attachment B2.

A9. Payment to Respondents

There is no payment to any respondents.

A10. Assurance of Confidentiality

A10a. Cohorts I and II Cross-site Evaluation

All information gathered through the administration of the CLI (Parts I and II) focuses on organizational activities undertaken as part of the SPF SIG program, rather than information about individuals. However, all respondents to the CLI (Parts I and II) will be required to register with the online survey site where the instrument will be completed. As part of this registration, it will be necessary to obtain identifying information about these individuals (i.e., name, e-mail address, organizational affiliation, and title/position). This information will be used for the creation of a user profile and every attempt will be made to keep this information private. After participants have registered with the Web site they will be provided with a UserID and temporary password to ensure that all of their survey responses remain private. Additionally, no survey responses will be attributed to a specific individual in any reports prepared from this data.

CLI (Parts I and II) participants will also be provided with the following information prior to completing the instrument: the purpose of the instrument; how the results will be used; the fact that participation is voluntary; that they may refuse to answer any question at any time or end the instrument at any time; that responses will be kept private to the extent possible; that individual names and positions will not be connected with any responses in any reports prepared from the data; and that all individual responses will be combined with the responses of others in all reports prepared from the data.

A10b. Cohorts III and IV Cross-site Evaluation

A10b1. Grantee- and Community-Level Instruments

All information gathered through the administration of the Grantee-Level Instruments (Infrastructure and Implementation) and Community-level Instruments (Parts I and II) focus on organizational activities undertaken as part of the SPF SIG program, rather than information about individuals. However, all respondents to the Grantee- and Community-Level Instruments) will be required to register with the online survey site where the instrument will be completed. As part of this registration, it will be necessary to obtain identifying information about these individuals (i.e., name, email address, organizational affiliation, and title/position). This information will be used for the creation of a user profile and every attempt will be made to keep this information private. After participants have registered with the website they will be provided with a UserID and temporary password to ensure that all of their survey responses remain private. Additionally, no survey responses will be attributed to a specific individual in any reports prepared from this data.

Grantee- and Community-Level Instrument respondents will also be provided with the following information prior to completing the instrument: the purpose of the instrument; how the results will be used; that responses will be kept private to the extent possible; that individual names and positions will not be connected with any responses in any reports prepared from the data; and that all individual responses will be combined with the responses of others in all reports prepared from the data.

A10b2. Participant-Level Instruments

Individual level data will be collected using the Participant-Level Instrument. As part of its grant application process, SAMHSA/CSAP requires that Cohort III and IV grantees describe the procedures they will use to ensure the privacy of participant data. These include by whom and how data will be collected, how data collection instruments will be administered, where data will be stored, who will/will not have access to information, and how the identity of participants will be safeguarded. Data files provided by the grantees to the DITIC do not contain client identifiers. The DACCC reviews these data files to ensure identifiers are removed before creating analysis files.

A11. Questions of a Sensitive Nature

No information of a sensitive nature will be directly collected on the Grantee- or Community-Level Instruments.

A12. Estimates of Annualized Hour Burden

The estimated annualized hour burden of conducting the cross-site evaluations of SPF SIG cohorts I and II and III and IV is 5,620.8 hours. How these hours were calculated is detailed in the sections below.

Table 1a. Estimates of Annualized Hour and Cost Burden to Respondents

Instrument Type	Respondent	Burden per Response (Hrs.)		No. of Respon-dents	No. of Responses per Respondent	Total Burden (Hrs.)	Hourly Wage Cost	Total Hour Cost
Grantee- Level Burden Cohort 1
Total/Average Burden Over 1 Reporting Year	Grantee	1		21	2	42	$42.00	$1,764
Community- Level Burden Cohort 1
CLI Part 1	Community	2.17		359	2	1,558.0	$32.00	$49,856
CLI Part 2	Community	2.17		359	6	4,674.2	$32.00	$149,574
Review of Past Responses	Community	2.5		359	2	1,795.0	$32.00	$57,440
Total/Average Burden Over 1 Reporting Year	Community					8,027.2	$32.00	$256,870
Grantee- Level Burden Cohort 2
Total Burden Over 2 Reporting Years	Grantee	1	5		4	20	$42.00	$840
Average Annual Burden	Grantee					10	$42.00	$420
Community- Level Burden Cohort 2
CLI Part 1	Community	2.17	86		4	746.5	$32.00	$23,888
CLI Part 2	Community	2.17	86		12	2,239.4	$32.00	$71,661
Review of Past Responses	Community	2.5	86		4	860.0	$32.00	$27,520
Total burden Over 2 Reporting Years	Community					3,845.9	$32.00	$123,069
Average Annual Burden	Community					1,923.0	$32.00	$61,536
Total Burden Cohorts 1 and 2
Total Burden Over 2 Reporting Years	Grantee					62	$42.00	$2,604
	Community					11,873.1	$32.00	$379,939
Average Annual Burden	Grantee					31	$42.00	$1,302
	Community					5,936.6	$32.00	$189,970
Grantee-Level Burden Cohorts 3 and 4
GLI Infrastructure & Implementation Instruments (Reporting Years 1-4)	Grantee	4.75		36	2	342.0	$42.00	$14,364
CLI Part I, 1—20: Community Contact Information (Reporting Year 1)	Grantee	1.5		36	1	54.0	$42.00	$2,268
CLI Part I, 1—20: Community Contact Information (Reporting Years 2-4)	Grantee	0.25		36	3	27.0	$42.00	$1,134
Total Burden Over 4 Reporting Years	Grantee					423.0	$42.00	$17,766
Average Annual Burden	Grantee					105.75	$42.00	$4,442
Community-Level Burden Cohorts 3 and 4
CLI Part I, 21–172: Community SPF Activities (Reporting Year 1)	Community	3		540	1	1620.0	$32.00	$51,840
CLI Part II (Reporting Year 1)	Community	0.75		540	6	2,430.0	$32.00	$77,760
CLI Part I, 21–172: Community SPF Activities (Reporting Years 2-4)	Community	0.75		540	3	1,215	$32.00	$38,880
CLI Part II (Reporting Years 2-4)	Community	0.5		540	18	4,860.0	$32.00	$155,520
Total burden Over 4 Reporting Periods	Community					10,125.0	$32.00	$324,000
Average Annual Burden	Community					2,531.25	$32.00	$81,000

Table 1b. Estimates of Average Annual Hour and Cost Burden to Respondents

	Respondent	Burden per Response (Hrs.)	No. of Respon-dents	No. of Responses	Total Burden (Hrs.)	Hourly Wage Cost	Total Hour Cost
Total Burden All Cohorts
Average Annual Burden	Grantee	1.75	62	69.5	121.3	$42.00	$5,093
	Community	1.08	985	5,107.5	5,499.60	$32.00	$175,985
	Overall	1.09	1,047	5,177.0	5,620.80		$181,078

A12a. Cohorts I and II Cross-site Evaluation

The estimated average annual burden for Cohort I and II grantee-level and community-level personnel is based solely on the completion of the Community Level-Instrument (CLI Parts I and II). Annualized reporting burden for 2 additional years of data collection for Cohorts I and II using the CLI (Parts I and II) is shown in Table 1a. The continuation of CLI data collection expects that Cohort I communities be required to complete the CLI for 1 additional year and Cohort 2 communities be required to complete the CLI for 2 additional years. Burden estimates are based on pilot respondents’ feedback as well as the experience of the instrument developers. Additionally, an individual community’s burden may be lower than the burden displayed in Table 1a because all sections of the Community-level Instrument (parts I and II) may not apply for each reporting period as community partners work through the SPF steps and only report on the step-related activities addressed. Note also that some questions will be addressed only once and the responses will be used to pre-fill subsequent instruments. To date, 359 communities have received SPF funds from their respective Cohort I States and 86 communities have received SPF funds from their respective Cohort II States. All of the directors of the community-based organizations that receive SPF funds are required to complete both parts of this instrument.

A12b. Cohorts III and IV Cross-site Evaluation

Estimates of total (across four years) and annualized reporting for Cohort III and IV grantee-level and community-level personnel are based on the completion of the Grantee-Level Instrument (GLI) and the Community-Level Instrument (CLI). Total and annualized burden estimates for grantee- and community-level personnel are displayed separately in Table 1a. Clearance to collect Participant-level NOMs outcomes is not being requested because no new burden associated with these evaluation components is being imposed. Specifically, the burden associated with the Participant-Level Instrument has already been approved by OMB for SPF SIG grantees (OMB No. 0930-0230).

The burden estimates for the GLI and CLI are based on the experience in the Cohort I and II SPF SIG evaluation, less the considerable reduction in length of these instruments implemented by the Cohort III and IV evaluation team. Burden estimates are provided by respondent group for specific instrument segments, by reporting year. In some cases, the burden estimates vary by year because all sections of the instruments may not apply for each reporting period as grantees and community partners work through the SPF steps and only report on the step-related activities addressed. In addition, some questions will be addressed only once and the responses will be used to pre-fill subsequent instruments.

Estimated burden of the grantee-level instruments is based on the current 16 grantees funded in Cohort III and an estimated 20 to be funded in Cohort IV, all of whom will be asked to complete the GLI Infrastructure and Implementation instruments twice each during the four year reporting period. Estimated burden of the community-level instruments assumes an estimated 540 communities (an average of 15 communities per grantee), annual completion of the CLI Part I, a minimum of two instrument updates per year for the CLI Part II, and an average of three distinct prevention intervention strategies implemented by each community during a 6-month period.

A13. Estimates of Annualized Cost Burden to Respondents

There are no capital/startup costs or operational/maintenance of services costs associated with this project.

A14. Estimates of Annualized Cost to the Government

The estimated annual cost to the Federal government of conducting the cross-site evaluations of SPF SIG cohorts I and II and III and IV is $552,257. How these costs were calculated is detailed in the sections below.

A14a. Cohorts I and II Cross-site Evaluation

The costs associated with the Cohorts I and II cross-site evaluation of the SPF SIG project, which is responsible for gathering, processing, analyzing, and reporting the data, serve as the basis for the estimated costs for these activities. The National Institute on Drug Abuse is funding all of the proposed activities. The estimated annual cost of the proposed extension is $446,070. Costs for 25 percent of a GS-14 CSAP project officer’s salary ($116,419 per year) of $29,104 have also been included. Thus, the total annual first year revision cost, associated with the evaluation of this project, is $475,174. The costs for the additional year would be reduced by 80% due to the decreased number of grantees (26 to 5). The average annual cost for Cohorts I and II is $285,104.

A14b. Cohorts III and IV Cross-site Evaluation

The estimated cost to the Federal government of conducting the evaluation of SPF SIG Cohort III and IV is based on the government’s contracted cost of the data collection and related evaluation activities along with the personnel cost of government employees involved in oversight and/or analysis. The DACCC is currently subcontracting with RMC for the majority of the data collection activities for which OMB approval is currently being requested, the annual cost to the government of this subcontract is $194,498. Additional costs include: 25 percent cost for a GS-14 CSAP project officer ($29,104), 10 percent cost for the DACCC Project Manager ($12,351), and 25 percent cost for the DACCC Data Manager ($31,200). Thus, the total annual cost to the government for this project evaluation is $267,153.

A15. Changes in Burden

Currently, there are 6,856 annualized burden hours in the OMB inventory for SPF SIG Cohorts I and II. CSAP is requesting 5,620.8 annualized burden hours for this revision. The number is shown in Table 1b. This decrease of 1,235.2 hours is due to the streamlining of evaluation instruments for the SPF SIG Cohorts III and IV Cross-site Evaluation.

For the continuation of data collection for Cohorts I and II, CSAP is requesting an average annual estimate of 5,967.5 hours for 26 grantees and 445 communities to complete the CLI instrument. The increase in burden specific to the continuation of data collection in Cohorts I and II is based on improved estimates for the number of interventions each community is likely to implement and thus need to report on. For the additional 36 grantees and 540 subrecipient communities included in the Cohort III and IV cross-site evaluation CSAP is requesting an average annual total burden of 2,637 hours to complete three survey efforts. The total annualized burden for the Cohort III and IV grantees and sub-recipient communities was reduced by changing the format of the grantee-level instruments from interview protocols to survey instruments, reducing the grantee-level data collection effort from annually to twice over the course of the funding period, and optimizing the use of skip patterns and replacing the majority of open-ended questions with multiple-choice-response questions in both the grantee- and community-level instruments.

A16. Time Schedule, Analysis and Publication Plans

This section describes the analysis, tabulation, and publication of results for the Cohorts I and II and Cohorts III and IV cross-site evaluations. The evaluation design for both sets of cohorts is similar except for the inclusion of a non-Cohort grantee comparison group in the Cohort I and II evaluation design and the addition of participant level data collection in the Cohort III and IV evaluation design. The following discussion pertaining to the evaluation schedule, analysis, and publication plans for Cohorts I and II and Cohorts III and IV will therefore be combined.

Research Questions

Eight outcome questions are guiding the SPF SIG outcome evaluation. Questions 4a and 4b referring to participant-level improvements are specific to Cohorts III and IV. These eight questions assess whether observed conditions/events can be attributed to SPF SIG programmatic interventions. The eight questions are:

1a. Did SPF funding improve grantee-wide performance on NOMs and other outcomes?

1b. What accounted for variation in NOMs and other outcomes performance across SPF grantees?

2a. Within grantees, did SPF funding lead to community-level improvement on NOMs and other outcomes?

2b. Within grantees, what accounted for variation in NOMs and other outcomes performance across funded communities?

3a. Across grantees, did SPF funding lead to community-level improvement on NOMs and other outcomes?

3b. Across grantees, what accounted for variation in NOMs and other outcomes performance across funded communities?

4a. Did SPF funding lead to participant-level improvement on NOMs and other outcomes?

4b. What accounted for variation in participant-level NOMs and other outcomes performance across funded grantees and communities?

In addition to these eight outcome research questions which are the central focus of the SPF SIG evaluation, the evaluation design also includes process-related research questions. These provide information necessary for interpreting the outcomes found in the evaluation, and focus on: interpreting the effects of project-related activities; identifying effective program and policy elements (e.g., conditions necessary for effective programs, populations for whom programs are effective); and assessing contextual factors related to SPF SIG outcomes. Some examples of process-related research questions included in the design are: What changes in allocation of funds and other resources for substance abuse prevention programs and other activities occurred at the grantee and community-levels; what grantee and community level mobilization and capacity building activities have been implemented; has cultural competence been integrated into prevention programs, policies, and practices in states, jurisdictions, and tribal entities; to what extent has the prevention infrastructure improved; to what extent are selected programs evidence-based; and to what extent are selected programs implemented with fidelity?

Logic Model of SPF SIG Impact

A logic model of SPF SIG impact has been developed to help guide the evaluation design and requirements. This logic model depicts the flow of grantee- and community-level activities that lead to systems change, and participant-level outcomes where evidence-based prevention-intervention programs are implemented. The model is depicted in Figure 1 below.

Grantee activities are represented in the logic model as rectangles, community activities are represented as ovals, and participant-level activities are represented as hexagons. The stacked ovals and hexagons represent the multiple communities and participants involved. The logic model is operationalized with SPF funding being received by funded jurisdictions and tribal entities. After receipt of funds, jurisdictions and tribal entities begin the planning and implementation of the SPF. The implementation of the SPF is expected to lead to both grantee-level systems change and funding of selected sub-recipient communities. Funding of selected communities is expected to lead to planning and implementation of the SPF also resulting in community-level system change. Systems change at both the state and community levels is expected to lead to changes in grantee- and community-level outcomes and participant-level outcomes.

The arrow connecting planning and implementation (both at the grantee and community levels) to systems change is bidirectional, indicating that both influence each other. The SPF model suggests that planning and implementation lead to systems change, and systems change leads to further refinement and efficiency of planning and implementation.

To examine the impact of SPF SIG funding on systems change and outcomes, the logic model includes variations in baseline status, contextual change and unmeasured factors for grantees and communities. Baseline status refers to pre-SPF SIG activities and achievements related to SPF SIG-initiated activities. Contextual change and unmeasured factors refer to anything that occurs in grantees and communities unrelated to the SPF SIG project that may potentially have an impact on systems change and outcomes.

Evaluation Timeline

Table 2 shows the time schedule for the cross-site evaluation of the SPF SIG initiative. The table is broken out to display two separate timelines specific to the Cohorts I and II and Cohorts III and IV evaluation schedules.

As indicated in Table 2 Cohorts I and II grantees began data collection of the CLI (Parts I and II) in January 2008, following OMB approval. Both parts of this instrument will be administered twice per year (every six months). Evaluation reports which include results of preliminary analyses conducted, using data from these instruments, have been produced every year in December. The first report was delivered in December 2005. A comprehensive final report for the SPF SIG will be delivered in September 2012.

Figure 1: SPF SIG Logic Model

Grantee-Level Epi Outcomes

Planning & Implementation

SPF $ in Selected Jurisdictions and Tribal Entities

Baseline Status

Grantee-Level

State-Level Systems Change

Contextual Change & Unmeasured Factors

LEGEND:

As also indicated in Table 2, all data collection and reporting requirements for Cohort III and IV are scheduled to begin following OMB approval (approximately July 2009) and end in September 2012. When funding for SPF SIG Cohort IV begins in FY2009, the reporting periods will be adjusted accordingly, depending on the length of funding for this cohort.

Table 2: SPF SIG Evaluation Time Schedule

Data Collection, Analysis, and Reporting Timeline
Evaluation Activity	Date
Cohorts I and II
Obtain OMB revision for CLI (Parts I and II)	TBD
Collect community subrecipient survey data (semi-annually)	Through July 2011
Analyze evaluation data to assess relationship between interview/survey data and outcomes	Annual interim analyses (2006-2011); comprehensive final analyses (2012)
Create data files for secondary analysis	December 2006 – December 2012
Produce bi-monthly reports	November 2004 – September 2012
Produce annual evaluation reports	December 2005 – December 2011
Produce final evaluation report	September 2012
Cohorts III and IV
Database Design	October 2008 –June 2009
Instrument Manual Development	November 2008- June 2009
OMB Package Under Review	May 2009- September 2009
Database Beta Testing	May –September 2009
Data Collection/Instrument Training	September 2009
Database Training	September 2009
OMB Package Approved	September 2009
PLI Reporting (bi-annually)	September 2009; November 2009 May 2010; November 2010 May 2011; November 2011 May 2012; September 2012
GLI Infrastructure Data Collection	September 2009 July 2012
GLI SPF Implementation Data Collection	September 2009 November 2011 (~ 36 months following approval of strategic plan)
CLI (Part 1)Data Collection	September 2009 March 2010 March 2011 March 2012
CLI (Part 2)Data Collection	September 2009; November 2009 March 2010; September 2010 March 2011; September 2011 March 2012; September 2012
Analyze evaluation data to assess relationship between survey data and outcomes	Annual interim analyses (2009-2011) Comprehensive final analyses (2012)
Annual Evaluation Reports	December 2009 December 2010 December 2011 December 2012 (Comprehensive Final Report)
Special Topic Reports	August 2010 August 2011

Plans for Tabulation and Analysis

Plans for tabulation and analysis will include qualitative and quantitative analyses of data collected and graphic and tabular displays of the key findings. The community-level epidemiological data and participant-level NOMs will be used to answer questions 1a, 2a, 3a, and 4a listed in section A16. The two grantee-level instruments (Infrastructure and Implementation) and Community-level Instrument (Parts I and II) will be used to gather data related to research questions 1b, 2b, 3b, and 4b listed in section A16, each of which addresses the effect of the SPF SIG initiative on grantee- and community-level system outcomes and community- and participant-level NOMs outcomes. Specifically, the four questions address the moderators and mediators of outcome variation across SPF-funded grantees and communities across funded grantees. Data from the Community-level Instrument (Parts I and II) in particular will be used to identify similarities and differences in the way SPF SIG is being implemented across communities and grantees including the specific prevention-intervention programs and strategy-types being implemented. This information will be linked to the participant-level NOMs for examining relationships between different types of community approaches for selecting and implementing evidence-based practices and corresponding outcomes.

Data reduction, scoring and scaling. The grantee-level Implementation and Infrastructure instruments and the CLI (Parts I and II) were developed using input from program staff in Cohort I and II states who are implementing the SPF initiative and policymakers who designed it. For Cohort III and IV grantees the grantee-level interviews were reformatted to be implemented in a survey format and the CLI (Parts I and II) were slightly modified. The original intent to include questions representing key aspects of prevention infrastructure and steps in the SPF initiative as outlined in the SPF-SIG RFA and GFA were maintained for all four instruments. Use of data from these instruments in outcome and process analyses will focus more on the scales and indexes that will be derived from each of the sections in the instrument than on a community’s or grantee’s responses to any individual item.

The first phase of the analysis of data from the grantee- and community-level instruments will consist of review, coding, scoring and scaling of responses within each instrument section with the goal of reducing the data to a set of reliable scales that will be used in subsequent analyses. For each section, summary scores or indexes will be developed that go beyond the limited response codes contained in the instrument to encompass the range of responses. Further development of empirically-based anchors for scales and the development of additional summary scores for sections will be based on analysis of the first wave of surveys using standard scale development procedures. Although considerable revision and winnowing of questions within sections has already taken place based on the pilot test, it is expected that some items in each section will yield more useful information for coding and some may show insufficient variation to be retained in final versions of the summary scores. Attention will be given to developing reliable and valid measures of the constructs in each instrument section, including relationships among both the items within potential summary scores and between the sections.

The collection of participant-level data was added to the Cohort III and IV evaluation to assess the impact of the SPF SIG on substance use and substance use related consequences. Responses to individual items collected from participants in direct-service programs lasting 30 days or longer will be used in combination with the grantee- and community-level instruments to examine the relationship between the SPF SIG process and participant-level outcomes.

Descriptive/normative analyses. Although the primary focus of the cross-site evaluation is on assessing impact, many descriptive and normative analyses will occur. The scales and indexes from the Grantee-level instruments and CLI (Parts I and II) will support these analyses, in tandem with coded data from archival sources such as grant applications, quarterly reports, and strategic plans. We will use standard techniques for analyzing, displaying, and reporting descriptive and normative results as they become available throughout the evaluation period. These will include summary statistics (means, medians, ranges, and standard deviations) and univariate and multivariate statistics (including cross-classification displays), as well as appropriate charts and graphs. Subsequently, the scales and indexes developed in the initial phases of analysis will also support the impact questions as key predictors of systems-, population-, and participant-level outcomes.

Outcome analyses. The data gathered will be used to conduct a variety of quantitative and qualitative analyses related to the eight outcome evaluation questions and also the process-related research questions. As part of these analyses, the distributional characteristics of the data as well as the baseline differences among the groups being compared will be assessed. Then, within-grantee and cross-grantee outcome analyses will be conducted using multilevel statistical modeling methods that account for the “nested” nature of the data (i.e., the data are not independent, they are nested within the communities and within the states). To estimate the effects of SPF, trends in repeated cross-sectional measurements of population (all cohorts) and participant-level (cohorts III and IV) outcomes at the grantee and community-level will be evaluated in these analyses. Additionally, propensity scores will be used to reduce potential bias from group nonequivalence between funded and non-funded communities in Cohorts I & II.

Statistical modeling methods will be performed using Hierarchical Linear Modeling (HLM) Version 6 (Raudenbush et al., 2002) ¹. The coefficients estimated by the HLM model are applicable to a hierarchical data structure with up to three levels of random variation. In our case, the three levels will be: 1) State, 2) community, and 3) time. HLM also accommodates sampling weights in both linear and nonlinear models. This is relevant to our analysis because 1) most of the NOMs and other outcomes will not meet normality assumptions and therefore require nonlinear models, and 2) states will contribute unequal numbers of communities and population sizes to the cross-site database. Therefore, inverse weighting by these inequalities at the appropriate level will increase the generalizability of the findings. Note that the Grantee-level instruments will support analyses of variation at level 3, the CLI will support analyses of variation at level 2, and both will support analyses of variation at level 1 through repeated administrations over time.

One system-level outcome of interest will be changes in prevention infrastructure over time. Data from the grantee-level instruments and CLI (Part I) will be used to measure state systems infrastructure. This includes changes in planning capacity, training capacity, and support for the implementation of evidence-based practices. Thus, data from these instruments will serve as outcome data for grantee systems change and as mediators of changes in population- and participant-level consumption and consequence outcomes. To support analyses that explain outcome variation among the SPF SIG grantees, a global index of grantee prevention infrastructure will be developed using data from the grantee instruments and CLI (Part I). This index will enable us to categorize the prevention infrastructure of grantees as “highly developed,” “moderately developed,” or “less well developed” over the course of SPF implementation. The grantee prevention infrastructure index will also be used in analyses to measure changes from year to year among the SPF SIG grantees.

The construct of prevention infrastructure is, however, too complex to be captured by a single summary statistic. In addition to the global index, therefore, indexes will also be developed based on specific infrastructure domains (planning, workforce development, etc.). Analyses of these indexes will help show whether some domains appear more critical to outcomes than others. Other analyses will focus on the relationship between SPF implementation and observed variation in outcomes across grantees.

Community-level analyses conducted with the data gathered from CLI (Parts I and II) will aim to identify characteristics of community-level interventions that are most effective in producing desired population and participant-level outcomes. These analyses will focus on: 1) comparisons of community-level outcomes from funded communities across multiple states with outcomes from unfunded communities where comparable data are available (Cohorts I & II only) or with state and national data; and 2) comparisons of systems-level outcomes across the funded communities, exploring the relationships between different types of community approaches, target populations, levels of implementation and fidelity, mix of strategy types, and aggregated community-and participant-level outcomes. Systems-level outcomes to be included in these analyses include changes in the number and operation of coalitions as assessed by CLI (Parts I and II). Population outcomes will focus on changes in consumption and consequences NOMs and other outcomes over time. Participant-level outcomes will focus on changes in risk perceptions and alcohol and drug consumption.

Public Use Data and Publication Plans

The cross-site team will provide CSAP with the reports necessary to determine, in consultation with the relevant federal staff, if the overall quality and quantity of the evaluation data are adequate for public release. Once it is determined that the data will be released, the cross-site team will perform a disclosure analysis of the data to detect both direct and indirect identifiers within the data, as well as the most likely sources for a possible breach of privacy. Based on the standards published by the Standing Review Committee for Disclosure Analysis at the Inter-University Consortium for Political and Social Research (ICPSR) the cross-site team will recommend a plan for each detected identifier. Once the disclosure plan is approved by CSAP, the cross-site team will produce a public use data file in compliance with ICPSR recommendations for public use data. Data will also be made available to the prevention community through the DITIC.

A17. Display of Expiration Date

The expiration date for OMB approval will be displayed on all approved instruments.

A18. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

B1a. Cohorts I and II Cross-site Evaluation

For Cohorts I and II, we are requesting a revision of the OMB approval for an existing data gathering instrument, the CLI (Parts I and II). Completion of this instrument will be required of all 26 active Cohort I and II SPF SIG grantees (states that receive SPF SIG funds) and an estimated 445 community subrecipients (communities that receive SPF SIG funds from the Cohort I and II grantees). The information gathered from this instrument will continue to be used by CSAP to monitor community subrecipients and state grantees and as an important data source for the Cohort I and II cross-site evaluation. The estimated response rate for this instrument is approximately 100 percent, as completion of this instrument will continue to be required from all Cohort I and II active community subrecipients and SPF SIG States.

B1b. Cohorts III and IV Cross-site Evaluation

SAMHSA’s CSAP accepted 16 competitive grant applications in FY 2006 and is accepting another estimated 20 applications for FY 2009 to fund SPF SIGs to States, Federally recognized Tribes, and U.S. Territories. Ultimately, SPF SIG grantees will assist and support selected community subrecipients (communities that receive SPF SIG funds from the Cohort III and IV grantees) to implement effective programs, policies and practices to reduce substance abuse and its related problems. Each grantee may pursue diverse strategies and methods for selecting the community subrecipients in their areas. It is estimated that an average of 15 community subrecipients will be awarded by each grantee.

Completion of the grantee-level instruments will be required of all active Cohort III and IV SPF SIG grantees. Completion of the Community-Level Instrument will be required of all active Cohort III and IV community subrecipients. The estimated response rate for these activities is approximately 100 percent, as completion will be required from all SPF SIG grantees and active community subrecipients.

Participant-level data will be required from all participants in direct-service programs lasting 30 days or longer. The participant-level instruments will be administered to each participant at program entry, program exit, and six months after program exit to examine the effect of direct service evidence-based strategies on participant-level NOMS outcomes.

B2. Information Collection Procedures

B2a. Cohorts I and II Cross-site Evaluation

Since this request is for of the continuation of the use of an existing instrument, the Cohort I and II cross-site team plans to continue existing data collection procedures as described below.

SPF SIG Project Directors in each Cohort I and II State and territory were initially contacted by the cross-site team by e-mail. This e-mail requested that the Project Director provide the name of a state-level administrator who has the option to review and approve completed CLIs (Parts I and II). A form was attached to this e-mail to be completed by the State-level administrator. This form requested contact information and was used to create a user profile in the web-based system. Once the form was completed and submitted to the Cohort I and II cross-site evaluation team, the state-level administrator was provided with a Username and Password and was granted access to the CLI (Parts I and II) Web site.

Once a state-level administrator was provided access to the CLI (Parts I and II) Web site, they were required to register the community agencies that have been awarded SPF funds in their state (i.e., subrecipient communities). All community subrecipients that receive SPF funding from their state were required to be registered in the web-based system. Each state-level administrator completed separate profiles for each of their subrecipient communities. Upon completion of the profiles, each community subrecipient was provided with a user profile in the web-based system and given access to the CLI (Parts I and II) website.

The CLI (Parts I and II) is to be completed every six months by all community subrecipients. Additionally, state-level administrators were given the option to review the information provided by the community subrecipients and were required to complete a brief set of nine questions. Reminder e-mails requesting completion of the CLI (Parts I and II) are sent at the end of each reporting period and two-weeks prior to the deadline for completing the instrument and on the due date. Community subrecipients and state-level administrators must complete the CLI (Parts I and II) within 45 days after the end of a reporting period. Follow-up e-mail reminders are sent the day after the deadline to state-level administrators and the communities who have not submitted all of their community subrecipient’s instruments to the Cohorts I and II cross-site evaluation team. A second follow-up e-mail is sent two weeks after the deadline to community subrecipients, state-level administrators, and state project directors for those community subrecipients who still have not completed the CLI (Parts I and II) and have not communicated the reasons for their delinquency to the cross-site team.

B2b. Cohorts III and IV Cross-site Evaluation

To initiate collection of the Grantee-Level Instruments the SPF SIG Project Director in each jurisdiction will be contacted by the Cohorts III and IV cross-site evaluation team by e-mail. This e-mail will request that the Project Director provide and or verify the name of the grantee-level evaluator who will be responsible for completing the two Grantee-Level Instruments. The grantee-level evaluator will be provided with a Username and Password and will be granted access to the website for completing the Grantee-Level Instruments.

A second e-mail will be initiated by the Cohorts III and IV cross-site evaluation team to request and/or verify the name and contact information of the person responsible (i.e., grantee-level reviewer) for completing Sections I–III of the Community-Level Instrument (Part I). Once a grantee-level reviewer has been provided access to the Web site, they will complete Sections I-III of the Community-Level Instrument (Part I) which includes registering the community agencies that have been awarded SPF funds in their jurisdiction (i.e., community subrecipients). All community subrecipients that receive SPF funding from the grantee are required to be registered in the web-based system. Each grantee-level reviewer will complete separate forms for each of their community subrecipients. Upon submission of the data to the Cohorts III and IV cross-site evaluation team, each community subrecipient will be provided with a user profile in the web-based system and provided access to the Community-Level Instrument (Parts I (Section IV-VIII) and II) on the Web site.

The grantee-level evaluators will complete both of the Grantee-Level Instruments twice over the span of the SPF SIG award. Grantee-level reviewers and community subrecipients receiving SPF SIG awards will be required to complete Part I of the Community-Level Instrument annually. Community subrecipients will be required to enter updates on Part II of the Community-Level Instrument a minimum of every six months. Reminder e-mails requesting completion of the Grantee-Level Instruments and Community-Level Instrument will be sent at one-month and two-weeks prior to the deadline for completing the instruments. Follow-up e-mail reminders will be sent the day after the deadline date to grantee-level evaluators and community subrecipients who have not submitted their instruments to the Cohort III and IV cross-site evaluation team. Federal Project Officers will be copied on all reminder e-mails sent prior to and after the deadline dates. A second follow-up e-mail will be sent two weeks after the deadline to Federal Project Officers and grantee Project Directors for those grantee-level evaluators or community subrecipients who still have not completed the instruments two weeks after the deadline notifying them of any outstanding instruments.

For the Participant-Level Instruments, each community subrecipient has its own plan for data collection, processing, data cleaning, control, and retention. Each plan describes how uniform data collection will be ensured and how participant protection will be assured. Participant-Level Instruments are administered by pencil and paper. Community subrecipients will submit data to the DITIC electronically, through CSAP’s CSAMS. Communities have been given the option to select only the Participant-Level Instrument measures that are relevant to the particular programs or strategies they are evaluating. When the community subrecipient logs on to CSAMS, they will be asked to confirm which measures they are using for each of their strategies. This will minimize data entry confusion, as only the appropriate measures will appear on their data entry screen for each strategy in the future. Communities will also be able to upload response databases through CSAMS that use the appropriate variable/value numbering. (Questionnaire codebooks will be made available on the CSAMS Website). SAMHSA/CSAP’s DACCC will be responsible for conducting logic checks on the data, and communicating with the grantees to clarify questions about the data.

Communities will be encouraged to enter Participant-Level Instruments as they are completed. At a minimum of every 6 months, communities will be required to ensure that all collected Participant-Level Instruments are entered. Reminder e-mails requesting updates to the Participant-Level Instrument data entry will be sent at one-month and two-weeks prior to the deadline for entering the instruments and again two weeks after the deadline. Federal Project Officers will be copied on all reminder e-mails sent prior to and after the deadline dates.

B3. Methods to Maximize Response Rates

B3a. Cohorts I and II Cross-site Evaluation

Because community subrecipients are required to complete the CLI (Parts I and II) the response rate has approached 100 percent for the last two rounds of data collection. The follow-up procedures, described in the preceding section, have been effective in the past and we expect to continue achieving a response rate between 98 and 100 percent for additional rounds of data collection, including those covered by this request.

B3b. Cohorts III and IV Cross-site Evaluation

Because grantees and community subrecipients will be required to complete the Grantee-Level Instruments and Community-Level Instrument (Parts I and II) as a condition of award, the response rate should approach 100 percent. The follow-up procedures, described in the preceding section, further increase the likelihood that a very high percentage of grantee-level evaluators, grantee-level reviewers, and community subrecipients will respond.

Communities will also be required to use the NOMs-based Participant-Level Instruments and enter them on the CSAMS website. Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for GPOs to identify problems and provide technical assistance. Because collection of the Participant-Level Instruments is a stipulation of the grants, it is anticipated that all communities will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents, and therefore communities cannot guarantee full cooperation on the part of participants. Historically, however, participant response rates across CSAP grantee sites have averaged 80 percent.

B4. Tests of Procedures

B4a. Cohorts I and II Cross-site Evaluation

Instrument Development

In the development of the CLI (Parts I and II), an extensive review of literature, program requirements, and evaluation frameworks was conducted to identify the appropriate concepts to measure. The following concepts were considered important to measure: community awareness of and openness to prevention efforts; relationship building, including coalition activities; organizational and community resources; sustainability; cultural competency; contextual factors; and systems and environmental factors.

State project directors, evaluators, and CSAP Federal Project Officers reviewed several versions of the CLI (Parts I and II). Their comments and suggestions on content and format were incorporated where appropriate. Additionally, the instrument was rigorously tested to ensure an appropriate reading level and was pilot tested with community grantees.

Pilot Testing of Instruments

The CLI (Parts I and II) was pilot tested in January 2006. Nine volunteers from four States participated in the pilot test. Pilot test participants were recommended by the SPF SIG Project Director in their State or their State’s evaluator, and represented the following types of organizations: mental health services; juvenile justice program services; substance abuse prevention services; youth-focused community organizations; and coalitions.

Pilot test participants provided feedback on the amount of time required to complete each part of the CLI as well as comments on the content of the instrument. Minor changes were made to both parts of CLI as a result of pilot testers’ feedback. These changes included: addition of definitions for specific terms used throughout the instrument; inclusion of examples of concepts; clarification of who should be answering specific questions (state-level administrator or community subrecipient); addition of response options; and addition of instructions to the instrument.

B4b. Cohorts III and IV Cross-site Evaluation

The two Grantee-Level Instruments and the Community-Level Instrument (Parts I and II) are modified versions of those used in the SPF SIG Cohort I and II Cross-site Evaluation (OMB No. 0930-0279). Earlier uses of these protocols did not identify any problems with questions being answered inappropriately and did not receive negative feedback from respondents about the topics covered. Modifications in the instruments resulted in a decrease in total burden by reorganizing the format of original instruments, optimizing the use of skip patterns, and replacing the majority of open-ended questions with multiple-choice-response questions. The modified instruments were revised by a team of evaluators with direct experience interacting with Cohort I grantees and community recipients who provided invaluable feedback from the perspective of the respondents. Grantee project directors, grantee-level evaluators, and CSAP Federal Project Officers reviewed the final versions and their comments and suggestions on content and format were incorporated where appropriate.

The core sections of the two Participant-Level Instruments will be drawn from the CSAP NOMs Adult and Youth Programs Survey Instrument Forms (OMB No. 0930-0230). The Adult and Youth Programs Survey Instrument Forms have been used numerous times for other CSAP funded studies. Subrecipient communities will have the opportunity to voluntarily select additional outcome measures that are relevant to their own initiatives. Grantee project directors, evaluators, and CSAP Federal Project Officers participated in previous meetings to select the menu of additional measures included in the two Participant-Level Instruments from a slate of validated instruments such as the Youth Risk Behavior Survey, National Survey on Drug Use and Health, Communities that Care Survey, etc.

B5. Statistical Consultants

B5a. Cohorts I and II Cross-site Evaluation

ETAG Consultants

Several individuals from the External Technical Advisory Group provided consultation on the statistical aspects of the evaluation design including:

Sandeep Kasat Epidemiologist PIRE 11710 Beltsville Dr., Ste 125 Calverton, MD 20705 Phone: 301-755-2745 Email: [email protected]

Wayne Harding, Ph.D. Social Science Research and Evaluation, Inc. 21-C Cambridge Street Burlington, MA 01803 Phone: 781-270-6613 E-mail: [email protected]

Cross-site Evaluation Staff

The primary individuals responsible for the analytic tasks for the evaluation of the Cohorts I and II SPF SIG initiative are:

Robert Orwin, Ph.D., the Cross-site Evaluation Principle Investigator, Westat, (301) 251-2277.

Bob Flewelling, Ph.D., Pacific Institute for Research and Evaluation, (919) 265-2621.

Additionally, several cross-site evaluation staff have expertise in statistical approaches to analyzing data and will also be contributing to the analytic tasks including:

Joseph Sonnefeld, M.A., Westat, (240) 214-2522.

Alan D. Stein-Seroussi, Ph.D., Pacific Institute for Research and Evaluation,

(919) 967-8998.

Agency Responsibility (SAMHSA/CSAP)

Within the agency, the following individual will have programmatic/evaluation oversight responsibility for Cohorts I and II:

John J. Park

Senior Public Health Analyst

Performance and Technical Assistance Branch

Division of Systems Development

US DHHS/SAMHSA/CSAP

1 Choke Cherry RD, Rm 4-1003

Rockville, MD 20857

B5b. Cohorts III and IV Cross-site Evaluation

Cross-site Evaluation Staff

The primary individuals responsible for the analytic tasks for the Cohorts III and IV SPF SIG Cross-site Evaluation are:

• Kelly Vander Ley, Ph.D., the Cross-site Evaluation Project Director, RMC Research Corporation, (800) 788-1887.

• Roy M. Gabriel, Ph.D., the Cross-site Evaluation Expert Advisor, RMC Research Corporation, (800) 788-1887.

• Virginia Mulkern, Ph.D., Project Director for the Data Analysis Coordination and Consolidation Center for SAMHSA's Center for Substance Abuse Prevention, Human Services Research Institute, (617) 876-0426.

Additionally, DACCC staff have expertise in statistical approaches to analyzing data and will also be contributing to the analytic tasks including:

• Nilufer Isvan, Ph.D., Data Analysis Team Lead for the Data Analysis, Coordination, and Consolidation Center project for SAMHSA's Center for Substance Abuse Prevention, Human Services Research Institute, (617) 876-0426.

Agency Responsibility (SAMHSA/CSAP)

Within the agency, the following individual will have programmatic/evaluation oversight responsibility for Cohorts III and IV:

Frank J. Winn

Social Science Analyst

Performance and Technical Assistance Branch

Division of Systems Development

US DHHS/SAMHSA/CSAP

 1 Choke Cherry RD, Rm 4-1013

Rockville, MD 20857

Attachments:

SPF SIG Cohorts I and II

A1 Instruments

1. CLI Part 1

2. CLI Part 2

A2 Instrument Reviewers

SPF SIG Cohorts III and IV

B1 Instruments

1. GLI Implementation

2. GLI Infrastructure

3. CLI Part 1

4. CLI Part 2

B2 Instrument Reviewers

1 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods, Second Edition. Newbury Park, CA: Sage.

SPF SIG Supporting Statement 27