Supporting Statement
B1a. Cohorts I and II Cross-site Evaluation
For Cohorts I and II, we are requesting a revision of the OMB approval for an existing data gathering instrument, the CLI (Parts I and II) and for the Sustainability Interview. Completion of these instruments will be required of all 26 active Cohort I and II SPF SIG grantees (states that receive SPF SIG funds) and, with respect to the CLI, an estimated 445 community subrecipients (communities that receive SPF SIG funds from the Cohort I and II grantees). The information gathered from these instruments will continue to be used by CSAP to monitor community subrecipients and state grantees and as important data sources for the Cohort I and II cross-site evaluation. The estimated response rates for these instruments is approximately 100 percent, as completion of these instruments will continue to be required from all Cohort I and II active community subrecipients and SPF SIG States.
B1b. Cohorts III, IV, and V Cross-site Evaluation
SAMHSA’s CSAP accepted 16 competitive grant applications in FY 2006, 25 applications for FY 2009, and 10 applications in FY2010 to fund SPF SIGs to States, Federally recognized Tribes, and U.S. Territories. Ultimately, SPF SIG grantees will assist and support selected community subrecipients (communities that receive SPF SIG funds from the Cohort III, IV, and V grantees) to implement effective programs, policies and practices to reduce substance abuse and its related problems. Each grantee may pursue diverse strategies and methods for selecting the community subrecipients in their areas. It is estimated that an average of 15 community subrecipients will be awarded by each grantee.
Completion of the grantee-level instruments will be required of all active Cohort III, IV, and V SPF SIG grantees. Completion of the Community-Level Instrument will be required of all active Cohort III, IV, and V community subrecipients. The estimated response rate for these activities is approximately 100 percent, as completion will be required from all SPF SIG grantees and active community subrecipients.
Participant-level data will be required from all participants in direct-service programs lasting 30 days or longer. The participant-level instruments will be administered to each participant at program entry, program exit, and six months after program exit to examine the effect of direct service evidence-based strategies on participant-level NOMs outcomes.
B2a. Cohorts I and II Cross-site Evaluation
The Cohort I and II cross-site team plans to continue existing CLI data collection procedures and, for the sustainability Interview, procedures developed of conducting the Phase I grantee-level data collection. SPF SIG Project Directors in each Cohort I and II State and territory were initially contacted by the cross-site team by e-mail. This e-mail requested that the Project Director provide the name of a state-level administrator who has the option to review and approve completed CLIs (Parts I and II). A form was attached to this e-mail to be completed by the State-level administrator. This form requested contact information and was used to create a user profile in the web-based system. Once the form was completed and submitted to the Cohort I and II cross-site evaluation team, the state-level administrator was provided with a Username and Password and was granted access to the CLI (Parts I and II) Web site.
Once a state-level administrator was provided access to the CLI (Parts I and II) Web site, they were required to register the community agencies that have been awarded SPF funds in their state (i.e., sub-recipient communities). All community subrecipients that receive SPF funding from their state were required to be registered in the web-based system. Each state-level administrator completed separate profiles for each of their sub-recipient communities. Upon completion of the profiles, each community sub-recipient was provided with a user profile in the web-based system and given access to the CLI (Parts I and II) website.
The CLI (Parts I and II) is to be completed every six months by all community subrecipients. Additionally, state-level administrators were given the option to review the information provided by the community subrecipients and were required to complete a brief set of nine questions. Reminder e-mails requesting completion of the CLI (Parts I and II) are sent at the end of each reporting period and two-weeks prior to the deadline for completing the instrument and on the due date. Community subrecipients and state-level administrators must complete the CLI (Parts I and II) within 45 days after the end of a reporting period. Follow-up e-mail reminders are sent the day after the deadline to state-level administrators and the communities who have not submitted all of their community sub-recipient’s instruments to the Cohorts I and II cross-site evaluation team. A second follow-up e-mail is sent two weeks after the deadline to community subrecipients, state-level administrators, and state project directors for those community subrecipients who still have not completed the CLI (Parts I and II) and have not communicated the reasons for their delinquency to the cross-site team.
The Sustainability Interview will be administered using the similar procedures employed during Phase I for the Infrastructure and Implementation Interviews. The National Cross-site Evaluation Team Interview Coordinator will contact the Single State Agency Director in each state and territory by email to provide an overview of the interview process and to elicit suggestions for potential respondents for the Sustainability Interview. Our respondents will be key stakeholders who have the knowledge and expertise to answer questions regarding the grantee’s prevention system in relation to steps of the SPF. Ideally, for each interview, the respondents will be the director of substance abuse prevention and the director of the Single State Authority (SSA) for substance abuse prevention and treatment. We will limit the interviews to 90 minutes to avoid overburdening the respondents but to allow enough time to capture detailed information and insights regarding the grantee’s infrastructure development. Each interview will be conducted by teams of two, consisting of a lead interviewer and a note-taker, both of whom will have considerable knowledge and expertise in prevention systems as well as experience conducting similar interviews. After each interview, the interview team will discuss the content of the interview to ensure they have a common understanding of the responses and will rate each grantee on its level of prevention capacity and infrastructure for each of the SPF steps (e.g., low capacity, moderate capacity, high capacity). The team will code and rate responses using scales included in the note-taking form Attachment A1a. Coded response data from the interview will be entered into an electronic database. Interview respondents will have the opportunity to request the interview notes from the interviewers (not the ratings) and to submit clarifying comments. Comments will be submitted within one week of receipt. Each interview team will review respondent comments and revise ratings as necessary.
Respondents also will be invited to send supporting documents electronically to further clarify responses which may include documents developed after SPF SIG funding expired, such as revised strategic plans; guidelines for use of ATOD data systems; written plans for developing a statewide ATOD prevention workforce; written policy/plan for addressing cultural competence in the state prevention system; guidelines for selecting/implementing culturally competent interventions; and evaluation/monitoring requirements for recipients of ATOD prevention funds.
B2b. Cohorts III, IV, and V Cross-site Evaluation
To initiate collection of the Grantee-Level Instruments the SPF SIG Project Director in each jurisdiction will be contacted by the Cohorts III, IV, and V cross-site evaluation team by e-mail. This e-mail will request that the Project Director provide and or verify the name of the grantee-level evaluator who will be responsible for completing the two Grantee-Level Instruments. The grantee-level evaluator will be provided with a Username and Password and will be granted access to the website for completing the Grantee-Level Instruments.
A second e-mail will be initiated by the Cohorts III, IV, and V cross-site evaluation team to request and/or verify the name and contact information of the person responsible (i.e., grantee-level reviewer) for completing Sections I–III of the Community-Level Instrument (Part I). Once a grantee-level reviewer has been provided access to the Web site, they will complete Sections I-III of the Community-Level Instrument (Part I) which includes registering the community agencies that have been awarded SPF funds in their jurisdiction (i.e., community sub-recipients). All community sub-recipients that receive SPF funding from the grantee are required to be registered in the web-based system. Each grantee-level reviewer will complete separate forms for each of their community sub-recipients. Upon submission of the data to the Cohorts III, IV, and V cross-site evaluation team, each community sub-recipient will be provided with a user profile in the web-based system and provided access to the Community-Level Instrument (Parts I (Section IV-VIII) and II) on the Web site.
The grantee-level evaluators will complete both of the Grantee-Level Instruments twice over the span of the SPF SIG award. Grantee-level reviewers and community sub-recipients receiving SPF SIG awards will be required to complete Part I of the Community-Level Instrument annually. Community sub-recipients will be required to enter updates on Part II of the Community-Level Instrument a minimum of every six months. Reminder e-mails requesting completion of the Grantee-Level Instruments and Community-Level Instrument will be sent at one-month and two-weeks prior to the deadline for completing the instruments. Follow-up e-mail reminders will be sent the day after the deadline date to grantee-level evaluators and community sub-recipients who have not submitted their instruments to the Cohort III, IV, and V cross-site evaluation team. Federal Project Officers will be copied on all reminder e-mails sent prior to and after the deadline dates. A second follow-up e-mail will be sent two weeks after the deadline to Federal Project Officers and grantee Project Directors for those grantee-level evaluators or community sub-recipients who still have not completed the instruments two weeks after the deadline notifying them of any outstanding instruments.
For the Participant-Level Instruments, each community sub-recipient has its own plan for data collection, processing, data cleaning, control, and retention. Each plan describes how uniform data collection will be ensured and how participant protection will be assured. Participant-Level Instruments are administered by pencil and paper. Community sub-recipients will submit data to the DITIC electronically, through CSAP’s PMRTS. Communities have been given the option to select only the Participant-Level Instrument measures that are relevant to the particular programs or strategies they are evaluating. When the community sub-recipient logs on to PMRTS, they will be asked to confirm which measures they are using for each of their strategies. This will minimize data entry confusion, as only the appropriate measures will appear on their data entry screen for each strategy in the future. Communities will also be able to upload response databases through PMRTS that use the appropriate variable/value numbering. (Questionnaire codebooks will be made available on the PMRTS Website). SAMHSA/CSAP’s DACCC will be responsible for conducting logic checks on the data, and communicating with the grantees to clarify questions about the data.
Communities will be encouraged to enter Participant-Level Instruments as they are completed. At a minimum of every 6 months, communities will be required to ensure that all collected Participant-Level Instruments are entered. Reminder e-mails requesting updates to the Participant-Level Instrument data entry will be sent at one-month and two-weeks prior to the deadline for entering the instruments and again two weeks after the deadline. Federal Project Officers will be copied on all reminder e-mails sent prior to and after the deadline dates.
B3a. Cohorts I and II Cross-site Evaluation
Because community sub-recipients are required to complete the CLI (Parts I and II) the response rate has approached 100 percent for the last two rounds of data collection. The follow-up procedures, described in the preceding section, have been effective in the past and we expect to continue achieving a response rate between 98 and 100 percent for additional rounds of data collection, including those covered by this request.
The Sustainability Interview will be administered after the SPF SIG funding has expired. Grantee key informants who participated in the Phase I interviews will not be obligated to participate. However, nearly all grantees participated voluntarily in multiple evaluation efforts during Phase I and we expect continued active participation in Phase II. We will work with the SSA to obtain contact information for respondents who are no longer involved in activities begun under SPF SIG. We expect a response rate approaching the 100 percent level.
B3b. Cohorts III, IV, and V Cross-site Evaluation
Because grantees and community sub-recipients will be required to complete the Grantee-Level Instruments and Community-Level Instrument (Parts I and II) as a condition of award, the response rate should approach 100 percent. The follow-up procedures, described in the preceding section, further increase the likelihood that a very high percentage of grantee-level evaluators, grantee-level reviewers, and community sub-recipients will respond.
Communities will also be required to use the NOMs-based Participant-Level Instruments and enter them on the PMRTS website. Issues related to response rates, as well as other data collection issues, are discussed at grantee meetings in order for GPOs to identify problems and provide technical assistance. Because collection of the Participant-Level Instruments is a stipulation of the grants, it is anticipated that all communities will comply (as appropriate). The participants at each site to whom these measures will be administered are all voluntary respondents, and therefore communities cannot guarantee full cooperation on the part of participants. Historically, however, participant response rates across CSAP grantee sites have averaged 80 percent.
B4a. Cohorts I and II Cross-site Evaluation
Instrument Development
In the development of the CLI (Parts I and II), an extensive review of literature, program requirements, and evaluation frameworks was conducted to identify the appropriate concepts to measure. The following concepts were considered important to measure: community awareness of and openness to prevention efforts; relationship building, including coalition activities; organizational and community resources; sustainability; cultural competency; contextual factors; and systems and environmental factors.
State project directors, evaluators, and CSAP Federal Project Officers reviewed several versions of the CLI (Parts I and II). Their comments and suggestions on content and format were incorporated where appropriate. Additionally, the instrument was rigorously tested to ensure an appropriate reading level and was pilot tested with community grantees.
The Sustainability Interview Guide is a synthesis of selected items from two interview protocols used during Phase I. The SPF Follow-Up Interview instrument has been adapted (and greatly reduced in length) from the SPF SIG Implementation and Infrastructure Interview guides used during Phase I of the cross-site evaluation. The SPF SIG national cross-site evaluation team was responsible for the development and pilot testing of both the SPF Implementation Interview and the State Infrastructure Interview instruments (OMB No. 0930-0279) from which the Sustainability Interview Guide was derived. Key prevention stakeholders, including state SPF SIG project directors and evaluators and other key SPF SIG staff, were consulted. They provided feedback on the content and format of the instruments’ domains, indicators, and measures to ensure that they had face validity and were not too burdensome for respondents to answer. In addition, all SPF SIG states were given the opportunity to review these instruments and provide comments and questions on their content and format. A draft of the 43 questions included in the Sustainability Interview Guide was completed in April 2010:
Pilot Testing of Instruments
The CLI (Parts I and II) was pilot tested in January 2006. Nine volunteers from four States participated in the pilot test. Pilot test participants were recommended by the SPF SIG Project Director in their State or their State’s evaluator, and represented the following types of organizations: mental health services; juvenile justice program services; substance abuse prevention services; youth-focused community organizations; and coalitions.
Pilot test participants provided feedback on the amount of time required to complete each part of the CLI as well as comments on the content of the instrument. Minor changes were made to both parts of CLI as a result of pilot testers’ feedback. These changes included: addition of definitions for specific terms used throughout the instrument; inclusion of examples of concepts; clarification of who should be answering specific questions (state-level administrator or community sub-recipient); addition of response options; and addition of instructions to the instrument.
The Sustainability Interview parent instruments—Implementation Interview and Infrastructure Interview guides (OMB No. 0930-0279)—were pilot tested in six states in October and November 2005. SPF/SIG Directors; State Epidemiology Work Group Chairs; State Advisory Committee members; SPF SIG evaluators; and SSA staff participated in the pilot testing. Two SPF SIG grantee evaluators who work with Cohort 1 and Cohort 2 grantees participated in a pilot study of the Sustainability Interview Guide. Participants confirmed that the amount of time needed to complete the interview did not exceed 90 minutes and also provided recommendations for clarifying the language of the questions.
B4b. Cohorts III, IV, and V Cross-site Evaluation
The two Grantee-Level Instruments and the Community-Level Instrument (Parts I and II) are modified versions of those used in the SPF SIG Cohort I and II Cross-site Evaluation (OMB No. 0930-0279). Earlier uses of these protocols did not identify any problems with questions being answered inappropriately and did not receive negative feedback from respondents about the topics covered. Modifications in the instruments resulted in a decrease in total burden by reorganizing the format of original instruments, optimizing the use of skip patterns, and replacing the majority of open-ended questions with multiple-choice-response questions. The modified instruments were revised by a team of evaluators with direct experience interacting with Cohort I grantees and community recipients who provided invaluable feedback from the perspective of the respondents. Grantee project directors, grantee-level evaluators, and CSAP Federal Project Officers reviewed the final versions and their comments and suggestions on content and format were incorporated where appropriate.
The core sections of the two Participant-Level Instruments will be drawn from the CSAP NOMs Adult and Youth Programs Survey Instrument Forms (OMB No. 0930-0230). The Adult and Youth Programs Survey Instrument Forms have been used numerous times for other CSAP funded studies. Sub-recipient communities will have the opportunity to voluntarily select additional outcome measures that are relevant to their own initiatives. Grantee project directors, evaluators, and CSAP Federal Project Officers participated in previous meetings to select the menu of additional measures included in the two Participant-Level Instruments from a slate of validated instruments such as the Youth Risk Behavior Survey, National Survey on Drug Use and Health, Communities that Care Survey, etc.
B5a. Cohorts I and II Cross-site Evaluation
ETAG Consultants
Several individuals from the External Technical Advisory Group provided consultation on the statistical aspects of the evaluation design including:
Sandeep Kasat Epidemiologist PIRE 11710 Beltsville Dr., Ste 125 Calverton, MD 20705 Phone: 301-755-2745 Email: [email protected]
|
Wayne Harding, Ph.D. Social Science Research and Evaluation, Inc. 21-C Cambridge Street Burlington, MA 01803 Phone: 781-270-6613 E-mail: [email protected]
|
Cross-site Evaluation Staff
The primary individuals responsible for the analytic tasks for the evaluation of the Cohorts I and II SPF SIG initiative are:
Robert Orwin, Ph.D., the Cross-site Evaluation Principle Investigator, Westat, (301) 251-2277.
Bob Flewelling, Ph.D., Pacific Institute for Research and Evaluation, (919) 265-2621.
Additionally, several cross-site evaluation staff have expertise in statistical approaches to analyzing data and will also be contributing to the analytic tasks including:
Joseph Sonnefeld, M.A., Westat, (240) 214-2522.
Alan D. Stein-Seroussi, Ph.D., Pacific Institute for Research and Evaluation,
(919) 967-8998.
Agency Responsibility (SAMHSA/CSAP)
Within the agency, the following individual will have programmatic/evaluation oversight responsibility for Cohorts I and II:
John J. Park
Senior Public Health Analyst
Performance and Technical Assistance Branch
Division of Systems Development
US DHHS/SAMHSA/CSAP
1 Choke Cherry RD, Rm 4-1003
Rockville, MD 20857
B5b. Cohorts III, IV, and V Cross-site Evaluation
Cross-site Evaluation Staff
The primary individuals responsible for the analytic tasks for the Cohorts III, IV, and V SPF SIG Cross-site Evaluation are:
Kelly Vander Ley, Ph.D., the Cross-site Evaluation Project Director, RMC Research Corporation, (800) 788-1887.
Roy M. Gabriel, Ph.D., the Cross-site Evaluation Expert Advisor, RMC Research Corporation, (800) 788-1887.
Virginia Mulkern, Ph.D., Project Director for the Data Analysis Coordination and Consolidation Center for SAMHSA's Center for Substance Abuse Prevention, Human Services Research Institute, (617) 876-0426.
DACCC External Steering Committee (ESC)
Several individuals from the ESC provided consultation on the statistical aspects of the evaluation design including:
Peter Mulhall, Ph.D.
Center for Prevention Research and Development
University of Illinois
Institute for Government and Public Affairs
510 Devonshire Drive
Champaign, IL. 61820
Phone: (217) 244-3231
E-mail: [email protected]
Diane Galloway, Ph.D.
College of Education
Department of Curriculum & Instruction
University of Texas-Arlington
1507 Clover Hill Road
Mansfield, TX 76063
Phone: (307) 421-9278
E-mail: [email protected]
Agency Responsibility (SAMHSA/CSAP)
Within the agency, the following individual will have programmatic/evaluation oversight responsibility for Cohorts III, IV, and V:
Beverlie Fallik, Ph.D.
DACCC COTR
Performance and Technical Assistance Branch
Division of Systems Development
US DHHS/SAMHSA/CSAP
1 Choke Cherry RD, Rm 4-1015
Rockville, MD 20857
Frank J. Winn, Ph.D.
DACCC SPFSIG cross site evaluation task lead
Performance and Technical Assistance Branch
Division of Systems Development
US DHHS/SAMHSA/CSAP
1 Choke Cherry RD, Rm 4-1013
Rockville, MD 20857
Attachments:
SPF SIG Cohorts I and II
A1 Instruments
Sustainability Interview Guide
CLI Part 1
CLI Part 2
A2 Instrument Reviewers
SPF SIG Cohorts III, IV, and V
B1 Instruments
GLI Implementation
GLI Infrastructure
CLI Part 1
CLI Part 2
| File Type | application/msword |
| File Title | RMC Basic Report & Proposal Template |
| Author | RMC Research Corporation |
| Last Modified By | summer.king |
| File Modified | 2011-05-02 |
| File Created | 2011-05-02 |