Cross-Site Evaluation of the Minority Substance Abuse/HIV Prevention Program
OMB Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) is requesting from the Office of Management and Budget (OMB) approval for the revision of data collection activities for the cross-site study of the Minority HIV/AIDS Initiative (MAI), which includes both youth and adult questionnaires. This revision includes the addition of 4 cohorts, changes to the data collection procedures based on intervention duration, and the addition of two questions on binge drinking behavior. The current approval is under OMB No. 0930-0298, which expires on 4/30/12.
This cross-site study supports two of SAMHSA’s 8 Strategic Initiatives: Prevention of Substance Abuse and Mental Illness and Data, Outcomes, and Quality. The primary objectives of the cross-site study are to:
Determine the success of the MAI in preventing, delaying, and/or reducing the use of alcohol, tobacco, and other drugs (ATOD) among the target populations.
Measure the effectiveness of evidence-based programs and infrastructure development activities such as: outreach and training, mobilization of key stakeholders, substance abuse and HIV/AIDS counseling and education, referrals to appropriate medical treatment and/or other intervention strategies (i.e., cultural enrichment activities, educational and vocational resources, and computer-based curricula).
Assess the process of adopting and implementing the Strategic Prevention Framework (SPF) with the target populations.
This program is authorized by Section 516 of the Public Health Service Act, as amended, and subject to the availability of funds. It was supported by the Congressional Black Caucus through its Conference Report on H.R. 4328, Making Omnibus Consolidated and Emergency Supplemental Appropriations Act, for FY 1998 (House of Representatives, October 19, 1998), to address prevention and treatment needs of minority communities that are disproportionately affected by HIV/AIDS. It builds on previously authorized programs addressing these issues (discussed below).
Although several Federal agencies have mandates to fund projects targeting minority and minority re-entry populations who are at risk for substance use and HIV/AIDS, very little is known about the efficacy of such programs once they become widely disseminated. Prior efforts to evaluate Federal substance use prevention initiatives targeting at-risk populations have focused on highly specific program models and strictly defined target populations or have been hampered by lack of valid instrumentation and poor study design. Although models have been disseminated to community-based agencies (that typically implement these programs under less rigorous and controlled parameters), measures and efforts to assess outcomes were inadequate and/or not sufficiently designed to determine the true impact of these interventions. In addition, the link between substance abuse and HIV/AIDS outcomes has not been evaluated for these programs or in local community settings.
Literature searches within SAMHSA’s three centers and with five other Federal agencies were conducted to identify studies with similar goals and expected outcomes. These searches have indicated that no similar study has been conducted which examines prevention initiatives regarding substance abuse (SA) in relation to HIV/AIDS. Even though the present cross-site study is unique from others that have been conducted in the field, information generated from these literature searches has sharpened the present cross-site study design and enhanced the likely utility of the results.
The cross-site study is scientifically appropriate, employs measures to safeguard the privacy and security of participants’ responses, and supports the program and study needs of multiple Federal agencies. Sample size, respondent burden, and intrusiveness have been minimized to be consistent with cross-site study objectives. To minimize and control respondent burden and to ensure the user-relevance of questions, every effort has been made to coordinate cross-site data collection with local data collection efforts, including pilot testing.
The cross-site study results will have significant implications for the substance abuse and HIV/AIDS prevention field, the allocation of grant funds, and other evaluation activities conducted by multiple Federal, State, and local government agencies. The results will be used to develop Federal policy in support of CSAP program initiatives, inform the public of lessons learned and findings, improve existing programs, and promote replication and dissemination of effective prevention strategies.
Background
Epidemiological studies on the dynamics of substance abuse and HIV/AIDS demonstrate a continued need to reach out to communities of color, particularly to those reporting high rates of HIV/AIDS and other sexually transmitted diseases (STDs). The Centers for Disease Control and Prevention (CDC) reported that although African Americans are only 13% of the U.S. population, they accounted for 52% of all HIV/AIDS cases diagnosed in 2008 and that Hispanics accounted for 19 percent of AIDS cases in 2008, despite making up only 15 percent of the U.S. population.
Of particular concern to communities of color is the return of ex-offenders, otherwise known as the re-entry population. Despite the efforts of correctional facilities to prevent sexual risk-taking behavior and substance abuse among incarcerated persons, a significant number engage in high-risk activities (such as IDU, tattooing, and coerced sexual activity), placing others at risk for HIV transmission. Each year, many of these persons, unaware of their HIV status, return to their communities and re-engage in substance abuse and other high-risk behaviors, putting others at an even greater risk for HIV/AIDS transmission.
Regardless of the mode of HIV transmission HIV/AIDS is an infectious disease that has drastic long-term medical, economic, and social consequences on minority populations. Meeting the challenges posed by HIV/AIDS requires close coordination with existing local, State, and territorial substance abuse and HIV/AIDS prevention programs. SAMHSA is working to improve access to quality services by increasing outreach and service capacity to at risk populations of color. Grantees are asked to use of the Strategic Prevention Framework as a method to prevent and reduce both substance abuse and the transmission of HIV/AIDS that will lay the necessary foundation for effective and sustainable prevention service delivery in the context of substance abuse and HIV/AIDS.
In FY 2009, the MAI Cohort 7 Program funded 55 five-year grants and in FY 2010 the MAI Cohort 8 Program funded 5 five-year grants to community-based organizations. These programs combined planning and services funding and required all grantees to participate in this cross-site study. They are expected to provide leadership and coordination on the planning and implementation of the SPF that targets minority populations and the minority re-entry population in communities of color with a high prevalence of substance abuse and HIV/AIDS. The primary objectives of the cross-site study are to:
Assess the process of adopting and implementing the SPF with the target population,
Measure the effectiveness of specified intervention strategies such as cultural enrichment, educational and evaluation activities, vocational resources, and/or computer-based curricula, and
Determine the success of the MAI in delaying, preventing, and/or reducing the use of alcohol, tobacco, and other drugs among the target population.
The MAI Cohort 9 and 10 programs build on previous SAMHSA/CSAP HIV/AIDS grant programs that provided substance abuse and HIV/AIDS planning and prevention services for minority populations.
The goal for Cohort 1 grants was to provide services. Goals for Cohorts 2-10 grants were to add, increase, or enhance integrated substance abuse and HIV prevention services by providing supportive services and by strengthening linkages among service providers for at-risk minority populations. HIV Cohorts 1-3 fell under SAMHSA/CSAP’s umbrella OMB Clearance Document 0930—0208. Since neither the HIV Cohort 4 nor the Cohort 5 Programs were cross-site studies, they did not require OMB clearance.
These past programs have enabled CSAP to make great progress in providing innovative, community-based drug prevention, planning, and intervention services to minority populations at risk for substance abuse and HIV/AIDS. HIV 6 grantees successfully increased perception of risk of harm and disapproval of peer use of alcohol and marijuana, attitudes that may influence behavioral change. The perception of risk of harm of alcohol and marijuana use both increased by 7.5 percentage points among youth, and 4.4 and 7.4 percentage points, respectively, among adults (Table 8).1. Of participants ages 12 to 20 who were non-users of alcohol at baseline 94.4% remained non-users at exit. Increasing knowledge of HIV and hepatitis may help prevent its transmission and reduce the stigma surrounding these two diseases. HIV Cohort 6 grantees also successfully increased knowledge of HIV and hepatitis, especially among youth. The percentage point change between baseline and exit for youth was 21.1% for the HIV Knowledge Scale and 34.6% for the Hepatitis Knowledge Scale.
The Cohort 6, 7, 8, 9 and 10 MAI Programs differ substantially from the earlier programs in that they target very different populations and call for the use of the SPF and evidence-based programs. While these grantees have substantial flexibility in choosing evidence-based programs, they are all required to base their projects on the five steps of SAMHSA’s SPF to build service capacity specific to substance abuse and HIV/AIDS prevention services.
The Minority HIV/AIDS Initiative (MAI) cross-site study will involve not only collecting information on the planning and delivery of the evidence based programs, but also assessing their effectiveness. Grantees will be conducting ongoing monitoring and analysis of their projects to assess program effectiveness, including Federal reporting of the 1993 GPRA, SAMHSA/CSAP NOMs, as well as HIV Counseling and Testing.
CSAP wishes to continue to enhance the nation’s impact on the HIV/AIDS epidemic with the current cross-site study, as data collection and analysis is designed to advance the current state of knowledge about the effectiveness of prevention programs for minority populations at risk for SA and HIV/AIDS, and to provide evidence and conclusions for disseminating optimally effective prevention policy and programs. Information collected will be used by CSAP and other Federal agencies in their efforts to assess specific intervention services in the prevention or reduction of substance use and HIV/AIDS among minority community and re-entry populations in communities of color across the nation. Information will also be useful to policymakers, who need to learn how to extend their reach into and among these populations.
CSAP will also share the outcome information and lessons learned with other Federal DHHS agencies, including but not limited to SAMHSA’s Center for Substance Abuse Treatment (CSAT), the National Institute on Drug Abuse (NIDA), the Centers for Disease Control and Prevention (CDC), and the Administration for Children and Families (ACF), which administers several drug-related programs targeted at hard-to-reach and at-risk populations.
Beyond DHHS, CSAP plans to share outcomes and lessons learned with:The Department of Justice (DOJ) and their Office of Juvenile Justice and Delinquency Prevention (OJJDP), which funds projects that target high-risk youth and often involve SA prevention interventions.
The Department of Housing and Urban Development (HUD), which supports two major initiatives (the National Youth Sports Program and the Public Housing Drug Elimination Program) that target youth at risk of substance use and provide positive alternative activities for at-risk youth in a drug-free environment.
The Department of Education (DOE), which funds the Safe Schools/Healthy Students effort (focused on violence and substance abuse prevention) under the Drug Free Schools and Communities Act.
State and local program planners and the public through publications and a public-use data set.
Implementing evidence-based programs in minority community settings presents challenges (i.e., maintaining rigor in design and instrumentation, as well as maintaining the ability to measure impact), given the need for local adaptations for specific target populations. Using the lessons learned from the previous programs, the current focus is on enhancing the effectiveness of specific interventions for reducing risk factors and/or enhancing the factors that protect against ATOD abuse and HIV/AIDS among the minority youth and adult population as well as the minority youth and adult re-entry population.
CSAP has a well-established history of incorporating evaluation findings and conclusions into the policy process, and the results of this study will be used similarly. It is designed to provide more specific information on the effectiveness of diversely funded programs in preventing and/or reducing ATOD use and related problems, CSAP will use the evaluation results to influence public policy, studies, and programming as they relate to the provision of youth and adult services. More specifically, the research will support the following uses by CSAP:
Findings will be used in required NOMs and GPRA performance reporting, and will be presented in annual reports to Congress.
Findings regarding SPF implementation will be used to assess prevention, delay or reduce ATOD use, influence positive sexual behaviors, change ATOD attitudes, and reduce associated problem sexual and substance use behaviors, as well as to assess the effectiveness of currently funded prevention programs. Furthermore, the common use of ATOD outcome measures (from CSAP, GPRA, and NOMs) will allow CSAP to compare initiatives (including the previous HIV/AIDS programs) as to their success in achieving their goals. Such extensive cross-initiative information will be used to set broad prevention policy priorities.
Findings concerning the ATOD and sexual behavior risk factors as both program outcomes and mediating factors will be used to refine policy and shape future program funding announcements. In addition, the findings may be used to provide recommendations to States regarding selection of evidence-based programs, since a portion of Block Grant monies given to the States must be spent on SA prevention. Additional monies have been awarded to some States through the Strategic Prevention Framework State Incentive Grants (SPF-SIG).
Findings concerning program inputs (intervention strategies, frequency, and length) will be used to provide program guidelines (e.g., through RFA’s) and to plan appropriate technical assistance services for programs/States.
Findings will support CSAP publications and materials on prevention practices that are an important resource for public and private organizations involved in the design and implementation of prevention programming for youth and adults.
In sum, the findings from the study will be a crucial resource for CSAP in setting prevention policy priorities, measuring performance, and designing and promoting optimally effective prevention program initiatives. Although the cross site study is designed primarily to address CSAP program requirements, the study results will be useful to other Federal, State, and community agencies involved in efforts to prevent or reduce ATOD use among youth and adults. While some of these agencies are specifically interested in providing preventive health services, others have a more general interest in approaches or strategies that have been proven effective.
State and local agencies also have significant responsibilities for design and implementation of prevention programs for youth and adults. The results of the MAI cross-site findings may be useful in a variety of ways to State and local agencies, including:
Policymakers in State and local governments will have evidence of the impact of various evidence-based programs and infrastructure development models on preventing or reducing ATOD use and HIV/AIDS among minority and re-entry youth and adults residing in communities of color. The evidence will be useful in setting prevention policy priorities.
Program planners in State and local governments and in community-based organizations will have comparative evidence on the effectiveness of different models for the provision of youth and adult services. This information will be useful in developing funding guidelines and direct service programs.
National, not-for-profit, nonprofit, voluntary, and professional organizations will have an accurate portrayal of the program inputs that are required to establish successful programs targeting minority and re-entry youth and adults residing in communities of color. This information will promote optimally effective prevention program design.
Changes
Data will be collected from approximately 47,241 respondents served by the 122 grantees at three time points (baseline or program entry, program exit, and three to six months post-exit) for interventions lasting 30 days or longer, two time points (baseline and exit) for interventions lasting between 2 and 29 days and one time point (exit only) for single session (one day) interventions. The CSAP National Outcomes Measures (NOMs) Adult and Youth questionnaires, which have been approved by OMB (OMB # 0930-0230) for use in all CSAP evaluation studies, will be used to measure ATOD use and risk factors associated with ATOD use among program participants. These NOMs data are used to report on Government Performance and Results Act (GPRA) and findings across CSAP programs.
For this program, these cross-site instruments are augmented with additional scales to measure other important risk and protective factors uniquely associated with HIV/AIDS among minority populations and minority re-entry populations in communities of color. The youth (covering ages 12-17) questionnaire contains 125 questions, of which 28 relate to HIV/AIDS and the adult questionnaire contains 118 items, of which 47 relate to HIV/AIDS. Two new questions have been added to both the youth and adult questionnaires to address SAMHSA’s need to collect information on binge drinking behavior, not covered under any prior OMB package.
These questions are:
Females only: During the past 30 days, on how many days did you have 4 or more drinks on the same occasion? [By 'occasion,' we mean at the same time or within a couple of hours of each other]
and
Males only: During the past 30 days, on how many days did you have 5 or more drinks on the same occasion? [By 'occasion,' we mean at the same time or within a couple of hours of each other].
Grantees will report these data online through the CSAP Prevention Management Reporting and Training System (PMRTS). Participants in interventions lasting 30 days or longer will be required to complete the entire questionnaire (three sections). Participants in interventions lasting between 2 and 29 days will complete sections one and two of the questionnaire and participants in single-session interventions will complete section one and 3-5 questions from section two. See breakdown below:
Intervention Duration |
Length |
Definition |
Sections of Survey To be Administered |
Single Session Intervention |
1 day or less |
A direct service intervention that lasts one day or less. Participants may receive multiple services during the session, but do not continue in a CSAP HIV grant funded activity for more than one day. |
|
Multiple Session Brief Intervention |
Less than 30 days |
The participant should receive at least two HIV Grant funded sessions or service encounters. The period of time between the first session or encounter and the last session or encounter should be two to 29 days. |
|
Multiple Session Long Intervention |
30 days or more |
The participant should receive at least two HIV Grant funded sessions or service encounters. The period of time between the first session/encounter and the last session/encounter should be 30 days or more. |
|
It is anticipated that technical infrastructure and data management skills will vary across grantee sites. To maximize data accuracy and reliability, online data entry tools will be designed for the instruments being submitted for clearance. These tools will be made available to grantees through CSAPS’s Prevention Management Reporting and Training System (PMRTS). The tools have been designed to reflect the structure of the instruments, and to allow the entry of data from completed survey forms directly into the system through the use of radio buttons corresponding to response options. The system will automatically quantify the selected response options and store the numeric codes in a SQL server for subsequent extraction, cleaning, and analysis.
The PMRTS is maintained by CSAP’s Data Information Technology Infrastructure Center (DITIC). The data entered online by grantees are periodically extracted by DITIC and transmitted in encrypted form to CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) for cleaning, record linkage, and analysis. Grantees have two options for accessing the data they entered online. In the first option, grantees can download the raw data they have entered online (as soon as it is submitted) in spreadsheet form. They can also access their data from the cleaned analysis files prepared by DACCC posted on PMRTS under password protection. In the second option, grantees can upload complete data files to the PMRTS. For this option, grantees are required to use a standard codebook while preparing the data, thus ensuring that uploaded data files have the same numeric coding and variable naming conventions as the data entered using the online tools.
These online data entry tools reduce the grantees’ burden by facilitating the data entry process and minimizing coding and variable naming errors. They also allow grantees without access to data management/analysis software to accurately quantify the information in completed survey forms. The DACCC will then conduct cross-site analyses to determine outcomes for the program as a whole.
The electronic multi-site data collection process will increase the efficiency and practical utility of the assessment of these programs. The CSAP multi-site questionnaires have been developed and used by grantees in previous HIV cohort programs and have been demonstrated to work efficiently and effectively. Based on the feedback of the HIV pilot, the questionnaires and procedures for electronic transmission of data files have been improved to increase efficiency and minimize burden on both training participants and grantee staff.
CSAP conducted an extensive literature search, consulted with staff in Federal agencies and organizations that work with ATOD and HIV/AIDS prevention programs, and discussed the proposed program with substance abuse prevention experts. Specifically, CSAP:
Conducted a comprehensive literature search of completed and ongoing studies of ATOD and HIV/AIDS prevention programs targeting youth and adults and found insignificant duplication with this cross-site study. All studies were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or application of results.
Consulted with staff in CSAT, CDC, NIAID, NIDA, ACF, OJJDP, HUD, DOE and DOJ. None of these Federal organizations has conducted a cross-site outcome evaluation of prevention and early intervention programs targeting minority youth and minority re-entry youth residing in communities of color that is similar to the one being proposed for this study.
Staff attended national meetings at which completed, ongoing, or contemplated evaluations were discussed and found insignificant duplication with the proposed study.
In summary, CSAP did not identify any redundancy in that there were no precedents for a cross-site study of projects similar to the one being proposed. Others suffered severe methodological problems such as low sample sizes, lack of dosage monitoring, and/or lack of published/reliable/valid measures and scales that make it unlikely that current information will be published or released among the scientific community or in respected journals, government publications, etc. Thus, it is clear that the data to be collected will be unique to the CSAP MAI programs, collected only for the CSAP programs, and not available elsewhere. The data collected through the multi-site effort will be non-duplicative, minimize burden on respondents, and be of use to both CSAP and the communities of color.
This data collection will have no significant impact on small entities.
The data will be collected from participants in interventions lasting 30 days or longer at three points in time: baseline (program entry), program exit and three to six months post-exit, two time points (baseline and exit) for interventions lasting between 2 and 29 days and one time point (exit only) for single session (one day) interventions. Failure to collect the information from all participants at all three points in time will result in missed opportunities and lessons learned on how to provide a quality improvement mechanism for CSAP to continually monitor and refine its prevention programs to ensure they meet the needs of minority populations and minority re entry populations at risk for SA/HIV/AIDS residing in communities of color. Data collected at all three points in time is essential for 30-day or longer interventions, as it will also demonstrate whether sustainable results can be achieved over time after the program has ended, and if so, for which types of interventions and populations.
Without this information:
CSAP will not be able to determine the extent to which it can prevent, reduce, and/or delay substance abuse and, in turn, reduce other risky behaviors that can lead to HIV/AIDS infection among minority populations and minority re-entry populations residing in communities of color.
CSAP will not be able to monitor the quality of its prevention programs and determine how they can be improved to ensure continued success at meeting the needs of minority populations at risk for HIV/AIDS.
CSAP will not be able to describe fully the range of prevention services used and the efficacy of evidence-based programs.
CSAP will not be able to ascertain if participants are more knowledgeable about HIV/AIDS and how they relate to SA as a result of program participation.
CSAP will not be able to identify those prevention services that are most effective and identify the potentially unique needs of minority populations residing in the community and minority re-entry populations.
CSAP will not be able to meet its Federal reporting requirements to DHHS, OMB, and Congress.
This information collection fully complies with 5 CFR 1320.5(d) (2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on July 27, 2011 (Vol. 76, page 44942). No comments were received.
The multi-site study and questionnaire design were based on initial consultation with SAMHSA experts from CMHS and CSAT, and on pilot testing with the previous HIV Cohorts. Other SAMHSA HIV program experts, such as Jenifer Fiedelholtz of OPPB and David R Robertson of CSAT (among others), were consulted on the following issues:
Draft study design plan and privacy/data security procedures; plan for coordinating and collecting data; measures to be used to assess outcomes; and mediating factors
Suitability of proposed assessment instruments
Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site study and reporting study findings
Means of minimizing the burden on project staff and program participants
Identification of efforts to ensure user relevance of results.
CSAP consulted with other experts on SA and HIV/AIDS, as well as other Federal agencies with related programs or mandates, including NIDA, ACF, CDC, DOJ, OJJDP, HUD, and the DOE. Consultations resulted in the refinement of measures and the coordination of Federal data needs.
No cash payment will be made to individual program participants from whom data will be collected.
CSAP has designed the multi-site questionnaire data collection strategy so that no identifying information such as names or Social Security Numbers will be requested of participants. Instead, each participant will be assigned a unique Individual Identifier number, as explained in the following paragraph.
To ensure privacy and security, grantees will assign each survey respondent a five-digit numeric code, or Individual Identifier. To further ensure the privacy and security of individual responses, all data will be reported at the aggregate level so that individual responses cannot be identified; no data will be reported at the individual participant level.
The data collection anticipated through this study falls under the Federal regulations regarding protection of human subjects in research (45 CFR 46). For all data collection concerning participating youth, informed written consent by parents and assent from the youth will be obtained. The evaluation will involve collecting data from local and re-entry minority youth and adults in communities residing in communities of color. They will complete a self-report questionnaire at three points in time, program entry, program exit, and three to six months post program exit. Several actions will be taken to protect their identities, including:
All data collected will be maintained in a safe and private manner. The DACCC and grantees will conform to all requirements of the Privacy Act of 1974 under the System of Records, Alcohol, Drug Abuse and Mental Health Epidemiologic Data, HHS/SAMHSA/OA, #09-30-0036.
Grantees will not send identifying information (i.e. name of respondent) to the DACCC. Only a questionnaire identification number will be provided. In addition, grantees will not provide identifying information to CSAP.
Access to the data will be limited to the DACCC staff directly involved in the evaluation. At the end of the grant, a public use data diskette or CD-ROM will be made available containing the HIV program grantees’ findings, along with detailed documentation. These public use data files will contain no individual identifiers. Reports prepared by the DACCC as contract deliverables will present data in aggregate form only.
All DITIC and DACCC staff will take a pledge agreeing that all information provided by respondents will be maintained with complete privacy and security.
Survey instruments include questions on ATOD use and attitudes, because these questions are necessary to obtain data that will help explain observed program outcomes. The proposed survey instruments incorporate all of the adult and youth NOMs items (OMB # 0930-0230) for GPRA and additional questions related to HIV/AIDS (OMB # 0930-0298). Since HIV/AIDS is a sexually transmitted disease, these survey instruments include a number of questions of a sensitive nature, such as questions on sexual behavior and practice and their relationship to SA. These questions must be asked because the interventions are focused on use of safe sexual practices, the use of condoms, and the reduction of unsafe sexual practices due to the contributing influence of ATOD use on acquiring HIV/AIDS.
Grantees will routinely obtain informed consent from parents of youth participating in the study. Written, informed consent will be a necessary prerequisite at every grantee site prior to data collection. Grantees will guarantee that all data submitted to the contractor for the DACCC has first received the appropriate written consent. This consent will also indicate data collection and release to the DACCC. These consent forms, unique to each grantee, will specify both the risks and benefits of study participation. There are no data elements in the data collection instruments covered by the consent that will fall outside of this protection.
Tables 1a and 1b show the estimated annualized burden for data collection. The evaluation data will be collected through questionnaires administered to youth and adult program participants. Youth and Adults in interventions lasting 30 days or longer will complete questionnaires three times, taking an average of 50 minutes for baseline, exit, and follow-up questionnaires. Participants in interventions lasting 2-29 days will complete questionnaires two times taking an average of 30 minutes to complete. Single-session intervention participants will complete one questionnaire at exit taking approximately 15 minutes to complete. Approximately 11,811 adults and youth are expected to respond annually across all 3 intervention types. Based on HIV Cohort 6 results, the expected response rates at exit are 62% and 37% at the three to six month follow-up in interventions lasting 30 days or longer. The total burden over the life of the grant is 59,375 hours and the average annualized burden is 14,844 hours for the four-year program evaluation study, as noted in the following table.
Table 1a. Estimates of Annualized Hour Burden by Intervention Length
Intervention Length |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Hour Burden |
Hourly Wage Cost |
Total Hour Cost |
30- Days or More Intervention |
|
|
|
|
|
|
|
Base line |
7,937 |
1 |
7,937 |
0.83 |
6,588 |
$8.00 |
$52,701 |
Exit |
4,887 |
1 |
4,887 |
0.83 |
4,056 |
$8.00 |
$32,446 |
Follow-up |
2,942 |
1 |
2,942 |
0.83 |
2442 |
$8.00 |
$19,538 |
Subtotal |
7,937 |
|
15,766 |
|
13,086 |
|
$104,685 |
2 to 29 Day Intervention |
|
|
|
|
|
|
|
Base line |
1,416 |
1 |
1,416 |
0.5 |
708 |
$8.00 |
$5,662 |
Exit |
872 |
1 |
872 |
0.5 |
436 |
$8.00 |
$3,486 |
Subtotal |
1,416 |
|
2,287 |
|
1,144 |
|
$9,149 |
Single Day Intervention |
|
|
|
|
|
|
|
Exit |
2,458 |
1 |
2,458 |
0.25 |
614 |
$8.00 |
$4,916 |
Annualized Total |
11,811 |
|
20,512 |
|
14,844 |
|
$118,749 |
Table 1b. Estimates of Annualized Hour Burden by Survey Type
Questionnaire |
Number of Respondents |
Total Responses |
Total Hour Burden |
Hourly Wage Cost |
Total Hour Cost |
Annualized Total Adult |
9,682 |
16,899 |
12,234 |
$8.00 |
$97,868 |
Annualized Total Youth |
2,128 |
3,612 |
2,610 |
$8.00 |
$20,881 |
Annualized Total |
11,811 |
20,512 |
14,844 |
|
$118,749 |
The burden estimate presented in Tables 1a and 1b is based on Cohort 6 findings. There will be no direct cost to youth or adults for participating in the study. The value of youth and adult time was assumed given the prevailing minimum wage rate in California (California was chosen since it is often the “bellwether” for setting precedents later adopted by other States).
There will be no capital, start up, or operation and maintenance costs incurred by the adults and youth participating in this study. The PMRTS on-line data collection tool will be available at no cost to grantees.
The total contract award for the DACCC will cover all aspects of the study design, planning, data collection, and analysis, with the annualized cost of $685,846. These costs cover the following activities: assistance to study sites in cooperation with the national evaluation; cleaning and processing of outcome data from study sites; data analysis and reporting; and development of public use data and documentation.
It is anticipated that the Government Project Officers who oversee the projects will expend time in assisting the DITIC and DACCC and grantees in appropriately responding to the measures. The GPO overseeing the HIV Cross-site will expend a portion of time overseeing the analysis of the HIV Cross-site data, as well as updating the data collection procedures for new cohorts. Data analysis activities include processing the data received from the various programs, as well as conducting statistical analysis. These costs are broken out in the table below. Annual hours are based on a 40-hour work week for 48 weeks per year. It is estimated that 8 CSAP Project Officers will be involved for approximately 2 percent of their time at an average annual salary of $110,000. The annualized total cost to the government will be $710,806.
Table 2: Estimated Annualized Cost to the Government
Position |
Percent FTE |
Annual Hours |
Rate |
Total Annual Cost |
DACCC Cross-site |
NA |
NA |
NA |
$685,846 |
HIV Cross-site GPO |
10% |
192 |
$40 |
$7,680 |
DITIC GPO |
2.50% |
48 |
$40 |
$1,920 |
HIV GPOs (Based on Estimated 8 GPOs) |
2.50% |
384 |
$40 |
$15,360 |
Total |
|
|
|
$710,806 |
Currently there are 16,770 hours in the OMB inventory. CSAP is requesting 14,844. The decrease of 1,926 hours is due to an adjustment. The original estimate was based on all participants being enrolled in 30-day or longer interventions. The current adjustment also accounts for interventions lasting less than 30 days.
Analysis Plans
The defining characteristic of this cross-site study is the sharing of a common protocol, a common set of performance measures, common outcome objectives, and common evaluation questions by all participating grantees. This study differs from more traditional multi-site clinical trials because each individual grantee will select Evidence Based Interventions (EBI) and Diffusion of Effective Behavioral Interventions (DEBI) that are adapted to the needs of the particular target population, setting, and organizational characteristics. This multi-site evaluation does not test a single intervention that has different settings, rather it is testing a category of interventions that have similar outcome objectives but that use different approaches to accomplish those objectives.
Analysis of a multiple-site data set requires a complex set of interrelated tasks. Planning for these tasks must be flexible, and must allow adjustments as the opportunities and challenges presented by the empirical realities of the data set are discovered. While multi-site studies provide strong opportunities for knowledge generation (because of the ability to contrast intervention and implementation variation in a single study), they also present significant evaluation challenges. This study recognizes those challenges and anticipates solutions as they will apply to the 122 participating grantees.
Sample Size Determination. Individual grantees have proposed their target population sizes. The establishment of sample size at the grantee level depends to some extent on financial constraints for program intervention services, staff allocation, participant retention activities and evaluation activities, including stipends.
Statistical Procedure Determination. As a multi-site design, the CSAP initiative collects information at two levels of observation: 1) across individuals, and 2) within individuals at three points in time for individuals in interventions lasting 30 days or longer and two points in time for individuals in interventions lasting between 2 and 29 days. The units of observation also have a hierarchical relation. Individual level units (youth and adults) are nested within program sites and points in time are nested within individuals.
The proposed analysis includes several distinct steps:
First, pooled analyses of outcomes will be conducted to assess the (controlled) presence of significant factors in growth curve trends for youth and adults participating in prevention interventions.
Second, the heterogeneity of outcomes across sites will be assessed to determine if outcomes for substance use or important protective factors significantly differ across sites. If there are significant differences, hypotheses will be developed to explain those differences and conduct multivariate analyses on: a) clusters of sites that share characteristics hypothesized to be contributors to effectiveness, and/or b) individual sites that exhibit combinations of principles and practices hypothesized as contributors to effectiveness.
Additional analyses will test the sensitivity of effectiveness models to differences in participant characteristics.
Statistical Test Determinations. Both the structural equation model (SEM) approach to estimating the trajectory parameters and the hierarchical linear model (HLM) approach that can consider time to be nested within an individual will be the key analytic methods conducted for this multi-site evaluation (MSE) (Bollen, 1989; Chou, Bender, & Pentz, 1998). This hierarchical data set presents flexible analysis opportunities as well as some analytic challenges. As the primary statistical tool, the plan is to apply multi-level regression [e.g., SEM and HLM] models. This technique allows for the identification of individual effects, controls for co-variates (e.g., propensity scores to control for non-equivalence across intervention groups), and tests for interaction effects with the different types of interventions and youth or adult characteristics that may mediate the impact of the intervention. HLM also provides excellent capacity for analyzing longitudinal, repeated measures designs (Willett, Singer, and Martin, 1998), can accommodate missing data at individual data points, and allow adjustments for different individual intervals between follow-up data points. Tests for attrition bias and selective attrition are conducted at each follow-up point.
A two-step analysis strategy will be used to control for attrition bias. First, baseline characteristics of all participants will be compared with baseline characteristics of participants who completed the programs. These two sets of data records will be compared with respect to baseline values of demographic characteristics, incarceration/reentry status, levels of risk and protection, and levels of substance use. Second, baseline factors found to differ significantly between the two sets of records will be included in all of the models and only model estimates net of these factors will be reported in the final evaluation results. This approach minimizes selectivity biases due to program attrition in reported program effects.
While only a few of the 122 program sites are incorporating a more robust experimental design with control/comparison groups, the MSE dataset will provide the flexibility to conduct analyses that provide useful evidence concerning the general effectiveness of prevention in reducing risk factors and promoting (developing) protective factors that could potentially affect SA or HIV/AIDS risk or transmission. The dataset will also be able to assess the variation in this effectiveness between interventions among the project sites, and offer potential explanations of that variation (e.g., amount of contact, type of intervention approach).
Given that target populations and community contexts vary by grant site, the multi-site dataset will contain data from a wide range of program participants in terms of their demographic, socioeconomic, and cultural characteristics. All multivariate models will include all of the factors that account for the differences among groups. Those factors that are found to have a significant effect on outcomes will be identified and interaction terms will be constructed to represent differences in program effects due to recruitment strategy. This analytic strategy will allow the evaluation study to take into consideration the mediating effects of a broad range of factors on program outcomes. The inclusion of these demographic, socioeconomic, and cultural control variables and interaction terms in the models will also ensure that final results are not biased toward the outcomes of groups with relatively large numbers of data records.
Many of the grantees are targeting African American and Hispanic/Latino populations and populations who have just been released (re-entry) from the criminal justice system. Youth between the ages of 12 and 17 and adults aged 18 and over are included in the cross-site study design. Varieties of methods are being used to recruit participants. Most sites report that participants will be identified from SA treatment programs, referrals from collaborations with criminal justice systems and other involved agencies, and/or through site-specific geographical areas that were identified through the SPF Step 1 needs assessment conducted during the first year of the grant. This needs assessment will identify key risk factors, including risk for substance use, HIV/AIDS transmission, and economic disadvantage. Step 2 will assess capacity to provide services. For Steps 3 and 4 (planning and implementation), grantees are allowed to select and adopt a variety of evidence-based prevention intervention approaches to fit the needs of their program participants/clients.
A multi-level analysis approach [e.g., SEM and HLM] will be used to investigate the effects of program characteristics on participant outcomes. Characteristics hypothesized to have a bearing on program effects, such as choice of prevention strategy and type of grantee organization, will be included in the dataset together with participant-level baseline, exit, and followup survey data. Nesting participant-level data within program-level data in this fashion will allow the construction of multi-level causal models that simultaneously test for the effects of participant and program characteristics on program outcomes and to identify significant interactions between these two levels.
Assessment of Exit-Only Data from Single-Session Interventions. The single-session data collection protocol requires administration of the questionnaire only at exit. Outcome analysis of these data will compare the exit responses of each participant with a norm constructed using the baseline responses to the item provided by a comparable group of participants from the same site who were in longer interventions (and hence, took both the baseline and exit surveys).
Publication Plans
The MAI cross-site study results will be made available to the public through publications and conference presentations. The following journals carry articles on SA prevention and HIV/AIDS and are expected to serve as potential vehicles for distribution of study results: Journal of Substance Abuse Treatment, International Journal of Addictions, Journal of Community Psychology, Journal of Adolescent Research, Journal of Adolescent Health, Preventive Medicine, Evaluation Review, Policy Studies Review, and the American Journal of Public Health. Study results could also be published in other journals that focus on HIV/AIDS. These include The Journal of the American Sexually Transmitted Disease Association, Health Education and Behavior, AIDS: Official Journal of the International AIDS Association, AIDS Education and Prevention, The Journal of Sex Research, AIDS Care, Psychological and Socio-Medical Aspects of AIDS/HIV, and Current Opinion in HIV and AIDS. Study results also are targeted for publication in journals focusing on infectious diseases. These include, among others, The Journal of the American Microbiological Association and Journal of Infectious Diseases.
The study results will be distributed through presentations at annual conferences of national and international public health organizations, such as the Society for Prevention Research, the American Public Health Association, the National Association of Alcohol and Drug Abuse Counselors, The National Prevention Network, the American Evaluation Association, and HIV/AIDS national meetings as well as regional and State SA prevention and treatment associations. HIV/AIDS meetings could include, among others, CDC Annual Conferences on AIDS and Conferences of the International AIDS Society. Results could also be presented at meetings focusing on infectious diseases such as annual meetings of the American Society of Microbiology.
Documents will also be prepared and published on behalf of the government (CSAP) through the Government Printing Office (GPO) for Federal agency and public use. Findings will also be available via OMB’s Website: www.expectmore.gov, as well as in annual reports to Congress and the performance detail sections of annual SAMHSA budgets as they become publicly available.
Timeline
The MAI is a five-year grant program (see Table 4). Years I and 2 are devoted to Steps 1, 2 and 3 of the SPF, namely conducting the needs assessment, capacity building, and planning, respectively. Years 3, 4, and 5 are devoted to Steps 4 and 5, implementation and evaluation, respectively.
Table 4. Project Timeline
Activity |
Cohorts 7 |
Cohort 8 |
Cohort 9 |
Cohort 10 |
Needs Assessment, Capacity Building, Planning |
FY 2009 |
FY 2010 |
FY 2011 |
FY 2011 |
OMB Clearance Obtained, Implementation |
FY 2011 |
FY 2011 |
FY 2011 |
FY 2011 |
Implementation |
FY 2010 |
FY 2011 |
FY 2011 |
FY 2012 |
Analysis and Reporting |
FY 2011 |
FY 2012 |
FY 2012 |
FY 2013 |
The expiration date will be displayed.
No exceptions are required.
REFERENCES
Bentler, P.M., (1995). EQS Structural Equations Program Manual. Encino: Multivariate Software, Inc.
Bollen, K.A. (1989). Structural equations with latent variables. New York: Wiley.
CDC 2010. HIV Surveillance Report: Diagnoses of HIV infection and AIDS in the United States and Dependent Areas, 2008, v.20.
Chou, C.P., Bentler, P.M., Pentz, M.A. (1998). Comparison of two statistical approaches to the study of growth curves: The multilevel model and latent curve analysis. Structural Equation Modeling, 5, 247-266.
Koziel, M.J., and Peters, M.C. (2007). Viral hepatitis in HIV Infection, M.D., NEJM 14: 356:1445-1454.
Springer, F. (2000). "Lessons learned from multiple cross-site evaluation studies.” Paper presented at the American Evaluation Association Meeting, Honolulu, November 2000.
Straw, R.B. & Herrell, J. M. (2002). Conducting multiple site evaluations in real-world settings. San Francisco, CA: Jossey-Bass.
Tedaldi, E.M. et al., (2003). Prevalence and characteristics of Hepatitis C coinfection in Human Immunodeficiency Virus Clinical Trails Group: The Terry Beirn Community Programs for Clinical Research on AIDS, Clinical Infectious Diseases, 15:36(10):1313-7
Willett, J.B., Singer, J.D., & Martin, N.C. (1998). The design and analysis of longitudinal studies of development and psychopathology in context: Statistical models and methodological recommendations. Development and Psychopathology, 10, 395-426.
Winstock, A.R., Anderson, C.M., and Sheridan, J. (2006). National Survey on HIV and Hepatitis Testing and Vaccination Services provided by Alcohol Agencies.
Minority
HIV/AIDS Initiative Page
| File Type | application/msword |
| File Title | HIV OMB Supporting Statement |
| Author | proddy |
| Last Modified By | DHHS |
| File Modified | 2011-09-28 |
| File Created | 2011-09-28 |