Cross-Site Evaluation of the Minority Substance Abuse/HIV Prevention Program
OMB Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) is requesting from the Office of Management and Budget (OMB) approval for the revision of data collection activities for the cross-site evaluation of the Minority HIV/AIDS Initiative (MAI). This revision includes 4 new cohorts, substantial revisions to the previously approved youth and adult questionnaires that reduced the number of items by at least twenty-five percent, and the addition of two brief forms for collecting program dosage data. The current MAI cross-site evaluation is approved under OMB No. 0930-0298, which expires on 2/29/2016.
The data collection instruments for which approval is being sought are:
Youth Questionnaire designed for respondents age 12 – 17 (revision);
Adult Questionnaire designed for respondents age 18 or older (revision);
Individual Dosage Form to keep records of services provided to each participant in one-on-one service encounters (new);
Group Dosage Form to keep records of services provided to participants in group settings (new).
This cross-site evaluation, currently being conducted by SAMHSA’s Program Evaluation for Prevention Contract (PEP-C), supports two of SAMHSA’s 6 Strategic Initiatives: Prevention of Substance Abuse and Mental Illness, and Health Care and Health Systems Integration, while also addressing the Healthy People 2020 Substance Abuse Topic Area HP 2020-SA.The primary objectives of the cross-site evaluation are to:
Assess the success of the MAI in preventing, delaying, and/or reducing the use of alcohol, tobacco, and other drugs (ATOD) among the target populations;
Assess the success of the MAI in reducing risk factors and increasing protective factors associated with the transmission of the Human Immunodeficiency Virus (HIV), Hepatitis C Virus (HCV) and other sexually-transmitted diseases (STD);
Measure the effectiveness of evidence-based programs and infrastructure development activities such as: outreach and training, mobilization of key stakeholders, substance abuse and HIV/AIDS counseling and education, testing, referrals to appropriate medical treatment, and other intervention strategies (i.e., cultural enrichment activities, educational and vocational resources, motivational interviewing & brief interventions, social marketing, and computer-based curricula);
Investigate intervention types and features that yield the best outcomes for specific population groups;
Assess the extent to which access to health care was enhanced for population groups and individuals vulnerable to behavioral health disparities residing in communities targeted by funded interventions;
Assess the process of adopting and implementing the Strategic Prevention Framework (SPF) with the target populations.
These objectives support the four primary goals of the National HIV/AIDS Strategy which include: 1) reducing new HIV infections, 2) increasing access to care and improving health outcomes for people living with HIV/AIDS, 3) reducing HIV-related disparities and health inequities, and 4) achieving a coordinated national response to the HIV epidemic.
This program is authorized by Section 516 of the Public Health Service Act, as amended, and subject to the availability of funds. It was supported by the Congressional Black Caucus through its Conference Report on H.R. 4328, Making Omnibus Consolidated and Emergency Supplemental Appropriations Act, for FY 1998 (House of Representatives, October 19, 1998), to address prevention and treatment needs of minority communities that are disproportionately affected by HIV/AIDS. It builds on previously authorized programs addressing these issues (discussed below).
Although several Federal agencies have mandates to fund projects targeting minority populations who are at risk for substance abuse and HIV/AIDS, our knowledge of the efficacy of such programs once they become widely disseminated is still limited. Prior efforts to evaluate federal substance use prevention initiatives targeting at-risk populations have focused on highly specific program models and strictly defined target populations or have been hampered by lack of valid instrumentation and poor study design. Although models have been disseminated to community-based agencies (that typically implement these programs under less rigorous and controlled parameters), measures and efforts to assess outcomes were inadequate and/or not sufficiently designed to determine the full impact of these interventions, especially on vulnerable populations. In addition, the link between substance abuse and HIV/AIDS outcomes has not been sufficiently evaluated for these programs or in local community settings.
The cross-site evaluation is scientifically appropriate, employs measures to safeguard the privacy and security of participants’ responses, and supports the program and evaluation needs of multiple federal agencies. Respondent burden and intrusiveness have been limited to the extent possible while providing sufficient data and analytic power to fulfill the cross-site evaluation’s objectives. To minimize and control respondent burden and to ensure the user-relevance of questions, every effort has been made to coordinate cross-site data collection with local data collection efforts.
The cross-site evaluation results will have significant implications for the substance abuse and HIV/AIDS prevention fields, the allocation of grant funds, and other evaluation activities conducted by multiple federal, state, and local government agencies. The results will be used to develop federal policy in support of SAMHSA program initiatives, inform the public of lessons learned from the findings, improve existing programs, and promote replication and dissemination of effective prevention strategies to further reduce health disparities.
Background
Epidemiological studies on the dynamics of substance abuse and HIV/AIDS demonstrate a continued need to reach out to communities of color, particularly to those reporting high rates of HIV/AIDS and other STDs. According to 2013 surveillance data from the U.S. and its six territories reported by the Centers for Disease Control and Prevention (CDC, 2015a), the rate (per 100,000 population) of HIV infection was 105.7 among Black/African Americans and 41.8 among Hispanic/Latinos, but only 13.8 among Whites.
Of particular concern to communities of color is the high level of HIV transmission among young people. According to CDC surveillance data, there were 40,634 individuals between the ages of 13 and 24 living with an HIV infection in the U.S. and its territories, as of the end of 2012. Of these adolescents and young adults, less than 15% identified themselves as White; the rest belonged to communities of color.
There are multiple psychosocial factors that render specific minority populations especially vulnerable to Substance Aabuse (SA) and HIV transmission, such as stigma, homophobia (experienced by gay, lesbian, bisexual, or transgender individuals), poverty, lack of health insurance, and lack of access to high-quality prevention and treatment services. Regardless of the mode of transmission, HIV/AIDS is an infectious disease that has drastic long-term medical, economic, and social consequences on minority populations. Meeting the challenges posed by this disease requires close coordination with existing local, state, and territorial substance abuse and HIV/AIDS prevention programs. SAMHSA is working to improve access to quality services by increasing outreach and service capacity to at-risk populations of color. Grantees are asked to use the Strategic Prevention Framework (SPF) as the foundation for effective and sustainable prevention service delivery in the context of substance abuse and HIV/AIDS.
According to the Institute of Medicine (IOM), Hepatitis C viral infection (HCV) is another growing concern in the United States (IOM, 2010). Cases of acute HCV cases have recently increased from 781 in 2009 to 2,138 in 2013. During the same period, the incidence rate more than doubled, increasing from 0.3 to 0.7 per 100,000. In 2013, individuals between the ages of 20 and 29 had the highest incidence rate with 2.01 acute cases per 100,000. When broken down by race and ethnicity, the rate in 2013 was highest among American Indians/Alaska Natives (1.7), followed by White non-Hispanics (0.82). The rates among Black non-Hispanics and Hispanics were 0.20 and 0.22, respectively (CDC, 2015b). In response to the recent emergence of HCV as a national public health concern, SAMHSA has included HCV testing, referral, and prevention efforts among activities required of the recent MAI cohorts, starting with grantees funded in 2014.
HIV and HCV share several risk factors, such as unprotected sexual intercourse and injection drug use (Garfein, Vlahov, Galai, Doherty, & Nelson, 1996; Terrault, 2002; Rodinelli et al., 2009). In addition to exacerbating the symptoms of viral Hepatitis through its adverse effects on liver functions, alcohol abuse also poses a risk for both HIV and HCV by increasing the likelihood of risky sexual behaviors and of noncompliance with treatment plans (Alter, 2002; Fisher, Bang, & Kapiga, 2007). Likewise, marijuana use has been shown to increase the incidence of risky sexual behaviors, especially among young adults (Brodbeck, Matter, & Moggi, 2006). These findings indicate a national need for integrated SA, HIV, and HCV prevention services. SAMHSA has responded to this need by structuring the requests for applications for MAI grants to require an integrated approach that combines SA, HIV, and HCV prevention interventions with HIV and HCV testing, counseling, and referrals for treatment.
Recent MAI Cohorts and Cross-Site Evaluation Results
This MAI cross-site evaluation builds on previous grant programs funded by SAMHSA's CSAP to provide substance abuse and HIV prevention services for minority populations. The first two cohorts were planning grant programs and the remaining were service grant programs. The goals for grants in Cohorts 3-6 were to add, increase, or enhance integrated SA and HIV prevention services by providing supportive services and strengthening linkages between service providers for at-risk minority populations. The HIV Cohort 1-3 cross-site previously received approval under OMB No. 0930-0208 and Cohorts 6 through 10 grants received approval under OMB No. 0930-0298. Since neither the HIV Cohort 4 nor the Cohort 5 programs were cross-site studies, they did not require OMB approval.
In FY 2009, the MAI Cohort 7 Program funded 55 five-year grants and in FY 2010 the MAI Cohort 8 Program funded 5 additional five-year grants to community-based organizations. These programs combined planning and services funding and required all grantees to participate in the MAI cross-site evaluation. Cohort 7 and 8 grantees were expected to provide leadership and coordination on the planning and implementation of the SPF and target minority populations and the minority re-entry criminal justice population in communities of color with a high prevalence of substance abuse and HIV/AIDS.
Two additional cohorts were funded in 2010. The Ready-to-Respond initiative (Cohort 9) awarded 34 five-year grants to former Cohort 6 grantees that had already built some prevention capacity and were expected to start implementing programs in expedited fashion. The Capacity Building Initiative (Cohort 10) funded an additional 27 grants to community-based organizations to build prevention capacity and to implement evidence-based interventions to prevent SA and HIV transmission in communities of color at high risk of HIV infection.
These past programs have enabled SAMHSA/CSAP to make great progress in providing innovative, community-based prevention, planning, and intervention services to minority populations at risk for substance abuse and HIV/AIDS. As exemplified by Tables 1 through 6 below, MAI grantees in Cohorts 7 through 10 successfully improved on a number of key outcomes. The tables below are reproduced from the most recent Minority AIDS Initiative (MAI) HIV Cross-Site Evaluation Report (December, 2013).
Table 1. Perception of Risk of Harm from Substance Use
Percentage of MAI Program Participants Ages 12 or Older Reporting Moderate or Great Risk of Harm from Having Five or More Drinks of Alcohol Once or Twice a Week and From Smoking Marijuana Once or Twice a Week at Baseline and Exit
Substance |
Valid N |
Baseline % |
Exit % |
Baseline to Exit Change |
Alcohol |
17,593 |
83.1 |
89.0 |
5.9** |
Marijuana |
16,642 |
66.9 |
76.7 |
9.7** |
Table 2. Disapproval of Peer Substance Use
Percentage of MAI Program Participants Ages 12 to 17 Who Somewhat or Strongly Disapprove of Their Peers Using Alcohol Nearly Every Day, Trying Marijuana and Using Marijuana Regularly at Baseline and Exit
Substance/ Target Group |
Valid N |
Baseline % |
Exit % |
Change |
Alcohol |
3,865 |
75.7 |
78.3 |
2.6* |
Trying Marijuana |
3,711 |
67.7 |
69.5 |
1.8 |
Regular Marijuana |
3,702 |
66.0 |
67.5 |
1.5 |
Table 3. Changes in Past 30-Day Substance Use
Average Days of Substance Use During the Past 30-Days by MAI Participants Ages 12 or Older at Baseline, Exit, and Follow-Up, by SAMHSA-Defined Target Group
Substance/ Target Group |
Valid N |
Average Days at Baseline |
Average Days at Exit |
Average Days at Follow-up |
Baseline to Follow-Up Change |
Exit to Follow-Up Change |
Alcohol |
|
|||||
Program Total |
5,522 |
4.2 |
3.2 |
2.8 |
-1.4** |
-0.4** |
Minority |
4,980 |
4.2 |
3.2 |
2.8 |
-1.4** |
-0.5** |
Minority Reentry |
825 |
5.6 |
4.4 |
4.0 |
-1.6** |
-0.4 |
Men Having Sex With Men (MSM) |
984 |
6.4 |
5.1 |
4.3 |
-2.0** |
-0.8** |
Black, Latina, or Hispanic Women |
2,792 |
3.6 |
2.6 |
2.1 |
-1.5** |
-0.5** |
Adolescents (Age 12 to 17) |
919 |
1.3 |
0.9 |
1.0 |
-0.3* |
0.1 |
Young Adults (Age 18 to 24) |
1,566 |
4.7 |
3.4 |
2.8 |
-1.9** |
-0.6** |
Older Adults (Age 50 and Over) |
911 |
4.6 |
4.0 |
3.7 |
-0.9** |
-0.2 |
Marijuana |
|
|||||
Program Total |
5,780 |
3.2 |
2.6 |
2.2 |
-0.9** |
-0.3** |
Minority |
5,210 |
3.1 |
2.5 |
2.2 |
-0.9** |
-0.3** |
Minority Reentry |
881 |
4.4 |
3.7 |
3.4 |
-1.0** |
-0.3 |
Men Having Sex With Men (MSM) |
1,045 |
3.9 |
3.3 |
3.0 |
-0.9** |
-0.4 |
Black, Latina, or Hispanic Women |
2,894 |
2.8 |
2.0 |
1.7 |
-1.0** |
-0.3* |
Adolescents (Age 12 to 17) |
969 |
2.9 |
2.2 |
2.1 |
-0.8** |
-0.1 |
Young Adults (Age 18 to 24) |
1,648 |
4.2 |
3.7 |
3.1 |
-1.1** |
-0.6** |
Older Adults (Age 50 and Over) |
941 |
1.8 |
1.5 |
1.1 |
-0.7** |
-0.3* |
Note 2: Figures within the Change column are calculated as the difference between follow-up and baseline and follow-up and exit. Minor discrepancies are due to rounding.
Table 4. HIV Knowledge
Average HIV Knowledge Scale Scores at Baseline and Exit for MAI Program Participants Ages 12 or Older, by SAMHSA-Defined Target Group
Target Group |
Valid N |
Baseline % |
Exit % |
Baseline to Exit Change |
Program Total |
19,777 |
70.9 |
81.8 |
10.9** |
Minority |
17,127 |
70.7 |
81.3 |
10.6** |
Minority Reentry |
2,546 |
73.3 |
83.0 |
9.7** |
Men Having Sex With Men (MSM) |
2,751 |
78.1 |
85.6 |
7.5** |
Black, Latina, or Hispanic Women |
9,348 |
71.2 |
82.6 |
11.4** |
Adolescents (Age 12 to 17) |
4,940 |
62.6 |
75.8 |
13.2** |
Young Adults (Age 18 to 24) |
6,011 |
74.2 |
83.4 |
9.1** |
Older Adults (Age 50 and Over) |
2,592 |
69.3 |
81.4 |
12.2** |
Note 3: The HIV Knowledge Scale consists of a battery of true/false questions about HIV transmission. The scale score is calculated as the percent of responses that are correct.
Table 5. Perception of Risk of Harm from Unprotected Sex
Percentage of MAI Program Participants Ages 18 or Older Reporting Moderate or Great Risk of Harm from Unprotected Sex at Baseline and Exit
Perceived Risk/ Target Group |
Valid N |
Baseline % |
Exit % |
Change |
Perception of Risk of Unprotected Anal Sex |
14,527 |
91.8 |
95.9 |
4.1** |
Perception of Risk of Unprotected Oral Sex |
14,549 |
78.1 |
86.6 |
8.5** |
Perception of Risk of Unprotected Vaginal Sex |
14,505 |
89.6 |
95.1 |
5.5** |
Table 6. Participants Reporting Protected Sex
Percentage of MAI Program Participants Ages 18 or Older Reporting That Their Last Sexual Encounter Was Protected at Baseline and Exit, by Type of Intercourse
Type of Sex/ Target Group |
Valid N |
Baseline % |
Exit % |
Baseline to Exit Change |
Protected Anal Sex |
1,112 |
48.8 |
60.9 |
12.1** |
Protected Oral Sex |
3,784 |
18.6 |
30.1 |
11.5** |
Protected Vaginal Sex |
4,369 |
37.0 |
50.9 |
13.9** |
Current Cohorts
At the time of this submission, Cohorts 9 and 10 are in the process of closing out their grants. Except for a few no-cost extensions, most of these grantees will conclude their activities by the end of FY 2015. In FY 2013, MAI’s Minority Serving Institutions (MSI) in Partnerships with Community-Based Organizations (CBO) Program funded 29 three-year grants to Historically Black Colleges and Universities, Hispanic Serving Institutions, American Pacific Islander Serving Institutions, and Tribal Colleges and Universities in partnership with CBOs in their surrounding communities to provide integrated substance abuse (SA), Hepatitis C (HCV), and HIV prevention services to young adults. In FY 2014, another cohort of 21 grantees was funded under the same program, and in FY 2015 SAMHSA funded an additional 34 MSI CBO grantees. Additionally, 54 five-year grants were funded in FY2015, under MAI’s Capacity Building Initiative (CBI). These grants will provide funds to community-level domestic, public and private nonprofit entities, federally recognized American Indian/Alaska Native Tribes and tribal organizations, and urban Indian organizations. CBI grants focus on building a solid infrastructure for integrated SA, HIV, and HCV prevention service provision and implementation of evidence-based prevention interventions. The target population for the CBI grantees will be at-risk minority adolescents and young adults.
While MAI grantees have substantial flexibility in choosing evidence-based programs, they are all required to base their projects on the five steps of SAMHSA’s SPF. This framework requires the grantees to build service capacity specific to substance abuse, HIV/AIDS, and for the more recent cohorts, HCV prevention services; to produce a strategic prevention plan based on local needs; and to implement appropriate evidence-based prevention programs.
Starting with Cohort 9 and continuing into the MSI CBO and CBI cohorts, the outcome focus of the MAI programs was broadened from individual-level change to include community-level change. The cross-site evaluation plan has therefore been revised to include an assessment of the grantees’ success in effecting population-level normative changes through the implementation of indirect service interventions such as media campaigns, social marketing, and environmental strategies. The data collection for which approval is being requested in this statement, however, is restricted to the individual-level data collection efforts of the evaluation. Approval will be requested in a separate statement for the instrument designed to collect community-level outcome data.
There will be four cohorts of grantees implementing the data collection protocol for which approval is being requested. Of these, the 85 grantees in the three MSI CBO cohorts are required to target young adults only; they are, therefore, not expected to use the youth instrument designed for the 12 – 17 age group. The 54 grantees in the CBI cohort being funded in 2015 have adolescents as part of their target population. In calculating burden, SAMHSA took into consideration that most of the evidence-based integrated SA/HIV prevention programs typically target adults. Among the past cohorts whose RFAs included youth among their target populations, only approximately one-third of the grantees implemented youth programs, with the rest providing prevention services to adults only. Based on this history, one-third of the 54 CBI grantees, that is, 16 grantees are expected to use the youth instrument. All 138 grantees are expected to use the adult instrument. Based on analysis of the data submitted by grantees in cohorts 7 – 10, it is estimated that the instruments will be used on a total of 25,944 participants per year. This estimate is for a typical year during which the grantee is implementing prevention interventions and will be zero during the start-up year when the grantee is going through the pre-implementation steps of the SPF.
The worksheet estimating the annualized numbers of participants expected to be included in the data collection effort, by the duration of services, is displayed in Exhibit 1. It is worth noting here that the duration of services is a key factor in burden estimation. As will be discussed in more detail later in this statement, the sections of the questionnaires administered to the participants vary by the length of time they spend in the funded programs. The number of dosage forms to be completed also varies by service duration.
Exhibit 1: Expected Numbers of Participants per Year, by Cohort and Service Duration
Service Duration |
MSI CBO Grantees |
CBI Grantees |
Total |
|||
Youth |
Adult |
Youth |
Adult |
Youth |
Adult |
|
1 day or less |
0 |
672 |
64 |
368 |
64 |
1,040 |
2 – 29 days |
0 |
2,772 |
240 |
1,542 |
240 |
4,314 |
30 days or more |
0 |
12,348 |
1,136 |
6,802 |
1,136 |
19,150 |
TOTAL |
0 |
15,792 |
1,440 |
8,712 |
1,440 |
24,504 |
Examples of How Data Collected through SAMHSA Minority AIDS Initiative have been Used
Cross-site evaluation findings have and will continue to be used for multiple purposes including: 1) to improve program performance, 2) meet statutory/regulatory requirements, and 3) contribute to extant knowledge about factors associated with substance use and HIV risk behaviors among minority populations.
In order to improve program performance, CSAP shares the most recent program performance data and related outcomes with grantees, stakeholders, and other Federal officials at MAI grantee meetings and webinars. These findings are then used to inform program direction and identify and address weaknesses.
SAMHSA/CSAP must collect these data to meet its federal requirements specified in the ADAMHA Reorganization Act of 1992 (PL 102-321), as well as the Government Performance and Results Act (GPRA) of 1993 and the GPRA Modernization Act of 2010 (PL 111-352). According to the ADAMHA Reorganization Act of 1992 (Section 516) “the Director of the Prevention Center, shall: (a) provide assistance to communities to develop comprehensive long-term strategies for the prevention of substance abuse; and (b) evaluate the success of different community approaches toward the prevention of such abuse. These requirements specify that SAMHSA evaluate this program, identify strengths and weakness, and assist the Director in making decisions necessary to strengthen the program and comply with this federal statute. In addition, data collected through the MAI cross-site evaluation help SAMHSA comply with federal reporting requirements related to appropriations (i.e. GPRA, NOMs, and the Department of Health and Human Services Core HIV Indicators). Currently, SAMHSA uses data collected through the MAI to report on five performance measures.
The MAI program findings on substance abuse and HIV risk behaviors have and continue to be widely disseminated through presentations and publications to inform the SA and HIV prevention research and practice communities. For the past nine years, CSAP has presented findings obtained from data collected through the MAI cross-site evaluation at national conferences held by organizations such as the National Prevention Network, the Society for Prevention Research, the American Evaluation Association, the American Public Health Association, and the American Psychological Association. These conferences are attended by leading researchers, practitioners, and policy makers in the field of prevention and program evaluation. In addition, using data from Cohort 3, CSAP and contract staff jointly published two peer-reviewed articles. One was a latent structure analysis of substance use and HIV risk behaviors among high-risk minority adults (Wang, et al., 2007) and the other was a structural equations analysis of HIV risk behaviors among sexually active minority adolescents (Bellamy et al., 2008). These presentations and articles exemplify SAMHSA’s continued contributions to the prevention field and to increased public awareness of factors associated with substance use and HIV risk behaviors among specific minority populations, through the use of data collected by MAI grantees.
SAMHSA will continue to contribute to the nation’s efforts to combat the HIV/AIDS epidemic by continuing the current cross-site evaluation of the MAI program. This includes data collection and analysis designed to advance the current state of knowledge about the effectiveness of prevention programs for minority populations at risk for SA and HIV/AIDS along with providing evidence and conclusions for disseminating optimally effective prevention strategies and programs. Information collected will be used by SAMHSA/CSAP and other federal agencies in their efforts to assess specific intervention services in the prevention or reduction of substance use and HIV/AIDS transmission among minority communities across the nation. Information will also be useful to policymakers in formulating programs and policies that impact substance use and HIV/AIDS transmission in communities of color.
SAMHSA will share the outcome information and lessons learned with other organizations within the Department of Health and Human Services (HHS), including but not limited to the National Institute on Drug Abuse (NIDA), the Centers for Disease Control and Prevention (CDC), and the Administration for Children and Families (ACF). These organizations administer several drug-related programs targeted at hard-to-reach and at-risk populations.
Beyond HHS, CSAP plans to share outcomes and lessons learned with:
The Department of Justice (DOJ) and their Office of Juvenile Justice and Delinquency Prevention (OJJDP), which funds projects that target high-risk youth and often involve SA prevention interventions.
The Department of Housing and Urban Development (HUD), which supports low-income persons and families living with HIV/AIDS through its “Housing Opportunities for Persons with AIDS Program.
The Department of Education (DOE), one of the collaborators in the multi-agency “Safe Schools/Healthy Students" effort (focused on violence and substance abuse prevention) under the Drug Free Schools and Communities Act.
State and local program planners and the public through publications, presentations, and a public-use data set.
Implementing evidence-based programs in minority community settings presents challenges such as maintaining rigor in design and instrumentation, as well as maintaining the ability to measure impact, given the need for local adaptations for specific target populations. Using the lessons learned from the previous programs, the current focus of the MAI program is on enhancing the effectiveness of specific interventions for reducing risk factors and/or enhancing the factors that protect against ATOD abuse and HIV/AIDS among the minority youth and young adult population.
SAMHSA/CSAP has a well-established history of incorporating evaluation findings and conclusions into its policy-making processes, and the results of this evaluation will be used similarly. More specifically, the data collection effort will support the following uses by CSAP:
Findings will be used in required National Outcome Measures (NOMs) and GPRA performance reporting and will be presented in annual reports to Congress.
Findings regarding SPF implementation will be used to assess the impact of evidence based programs on preventing, delaying or reducing ATOD use, changing attitudes toward SA, and in reducing problem sexual behaviors, as well as to assess the effectiveness of currently funded prevention programs. Furthermore, the use of common ATOD outcome measures will allow CSAP to compare initiatives (including the previous HIV/AIDS programs) as to their success in achieving their goals. Such extensive cross-initiative information will be used to set broad prevention policy priorities.
Findings concerning SA and HIV transmission risk factors as both program outcomes and mediating/moderating factors will be used to refine policy and shape future program funding announcements. In addition, the findings may be used to provide recommendations to states, territories, and tribal organizations regarding the selection of evidence-based SA prevention programs for specific vulnerable population groups targeted by grant programs such as the Strategic Prevention Framework Partnerships for Success.
Findings concerning the impact of program inputs utilized by the MAI grantees (intervention strategies, frequency, and length) on outcomes will be used to provide program guidelines (e.g., through Requests for Applications [RFA]) and to plan appropriate technical assistance services for grantees.
Findings will support SAMHSA publications and materials on prevention practices that are an important resource for public and private organizations involved in the design and implementation of prevention programming for youth and adults.
In sum, the findings from the MAI cross-site evaluation will be a crucial resource for SAMHSA in setting prevention policy priorities, measuring performance, and designing and promoting optimally effective prevention program initiatives. Although the cross-site evaluation is designed primarily to address CSAP program requirements, evaluation results will be useful to other federal, state, and local agencies involved in efforts to prevent or reduce SA and HIV/HCV transmission among youth and adults. While some of these agencies are specifically interested in providing preventive health services, others have a more general interest in approaches or strategies that have been proven effective.
State and local agencies also have significant responsibilities for the design and implementation of prevention programs for youth and adults. The results of the MAI cross-site findings may be useful in a variety of ways to these agencies, including:
Policymakers in state and local governments will have evidence of the impact of various programs and infrastructure development models on preventing or reducing SA, HIV/AIDS, and HCV among minority youth and young adults. The evidence will be useful in setting state and local prevention policy priorities.
Program planners in state and local governments and in community-based organizations will have comparative evidence on the effectiveness of different models for the provision of youth and adult services and for reducing behavioral health disparities. This information will be useful in developing funding guidelines and direct service programs.
National, nonprofit, voluntary, and professional organizations will have an accurate portrayal of the program inputs that are required to establish successful programs targeting minority youth and adults. This information will promote optimally effective prevention program design.
College administrators will gain an understanding of the strategies that best address the risk factors associated with SA and HIV transmission on campuses and the surrounding communities. This knowledge will help formulate policies for reducing illegal substance consumption and risky sexual behaviors among young adults.
Changes in Instrumentation
The youth and adult questionnaires currently in use (OMB # 0930-0298) have been substantially revised and shortened for the continuation of this cross site and this new OMB approval request. The current versions included all of SAMHSA’s prevention NOMs items, a requirement at the time the instruments were designed. Recently, this requirement has been relaxed and grant programs are now allowed to select only the NOMs items that are directly related to their targeted outcomes. Therefore, the first phase of the revision process was to eliminate the previously required NOMs items that are not closely associated with MAI’s goals. Additionally, multiple items measuring the same or very similar constructs were streamlined. The second phase of the revision process consisted of identifying a few new items that address SAMHSA’s newly emerging priorities, such as reducing behavioral health disparities, providing culturally and linguistically appropriate services, and assessing the impact of substances whose prevalence has increased in recent years (e.g. synthetic marijuana, electronic vapor products, and non-medical use of prescription drugs). Care was taken to ensure that the psychometric properties of the newly-added items had been established by federal agencies or other credible organizations responsible for large-scale epidemiological data collection efforts. The revised questionnaires for which approval is being requested contain 94 (youth) and 79 (adult) items in comparison to the lengths of the current instruments at 128 and 122 questions, respectively. The revisions to the current questionnaires are summarized in Exhibit 10, Section B2.
Participants whose services last 30 days or longer will be required to complete the entire questionnaire (three sections) at baseline (within 30 days before the first service encounter), exit (within 10 days after the last service encounter), and three-to-six-month post-exit follow-up. Participants whose services last between 2 and 29 days will complete sections one and two of the questionnaire at baseline and exit, and participants who are in the program for a single day will complete Section One and 3-5 relevant questions from Section Two at program exit only. Participants are free to refuse to respond to the entire survey or to particular items in the survey. A full description of the data collection protocol is provided in the Overarching Administration Guide (Attachment 5).
All questionnaires will be completed in self-reported paper-and-pencil format. Completed survey forms will be submitted to SAMHSA through the PEP-C data submission system, currently under development. The system is being designed to provide the grantees with data entry screens that mimic the paper questionnaires in order to facilitate data entry, reduce burden, and minimize data entry errors. An online data validation feature will check for inconsistencies and out-of-range values and will issue error messages during data entry. Grantees will have the option of preparing databases for batch upload, using standard coding manuals and templates developed by the PEP-C team. A data scrubber is under development for validating uploaded batch data files and issuing error messages to flag formatting errors, inconsistencies, and out-of-range values.
Based on analysis of data from Cohorts 7 through 10, not all participants take all of the required numbers of surveys. The burden estimates in this statement are based on analysis of these data and are adjusted for the proportion of participants who respond to the required surveys and to the distribution of participants among the three service duration categories.
In addition to the revised questionnaires, SAMHSA is seeking approval for two brief Dosage Forms for recording the service types and program dosage that each participant receives in direct service interventions. These will be completed by the grantee organization after each service encounter with each participant, except in cases where the only funded service that the participant receives is HIV or HCV testing and referral. In those cases, no dosage records will be required. Data on numbers tested and referred by the grantee will be collected by an online quarterly progress reporting tool for which OMB approval is being requested through a separate statement.
Separate Dosage Forms are available for recording one-on-one and group-format encounters with program participants. The Individual Dosage Form will be used to record one-on-one service encounters such as individual counseling and motivational interviewing and contains fields for entering:
Date of encounter;
Unique participant identification number;
Up to four service codes and number of minutes of exposure to each.
The Group Dosage Form will be used to record group-format services such as group counseling or health education classes and contains fields for entering:
Date of encounter
Up to three service codes and number of minutes of exposure to each.
Unique participant identification numbers of up to 80 participants
The list of service codes is included in the blank forms to reduce the grantees’ burden in entering this information.
Service providers will initially complete the paper dosage forms as part of their record-keeping process. The data will be submitted to SAMHSA through the PEP-C data submission system, currently under development. As with survey data, the system is being designed to provide the grantees with data entry screens that mimic the paper dosage forms in order to minimize burden and maximize the quality of the submitted data. As with survey data, grantees will have the option of preparing dosage databases for batch upload.
It is anticipated that technical infrastructure and data management skills will vary across grantee sites. To maximize data accuracy and reliability, online data entry tools will be designed for the instruments being submitted for approval. These tools will be made available to grantees through CSAPS’s Program Evaluation for Prevention Contract (PEP-C). The tools are being designed to reflect the structure of the instruments, and to allow the entry of data from completed survey and dosage forms directly into the system through the use of numeric fields and radio buttons corresponding to responses or response options. Grantees will also have the option to prepare their own databases for batch upload to the PEP-C system. For this option, grantees are required to use standard codebooks and database templates while preparing the data, thus ensuring that uploaded data files have the same numeric coding and variable naming conventions as the data entered using the online tools. The system will automatically quantify the selected response options and store the numeric codes in a SQL server for subsequent extraction, cleaning, and analysis. Grantees will be able to save the data they enter into a temporary area of the database. Grantee staff will be able to review and edit the temporary entries. Temporarily saved data will not be extracted for processing. Once the grantee has finalized their data entries, they will move them from the temporary area into the SQL database by using a “Submit” button. Submitted data are no longer available for editing by the grantees until the Cross-Site Team has extracted and reviewed them for quality. After the initial review, grantees will be sent a data validation report flagging records where issues were discovered. They will then be able to access their submitted data for editing in response to the validation report, for a period of two weeks. At the end of the two-week editing period, the PEP-C’s Systems Team will extract the data edited by the grantees second time, and transmit the extracted database to the MAI Cross-Site Evaluation Team for record linkage, cleaning, and analysis. Record linkage will involve locating and linking together all survey and dosage records submitted for each grantee into a single participant data record, using the participant’s unique identification number as the linkage key.
Grantees will have two options for accessing the data they entered online. In the first option, grantees will be able to download the raw data they have entered online (as soon as it is submitted) in spreadsheet form. They will also be able to access their data from the cleaned analysis files prepared by PEP-C’s MAI Cross-Site Team and posted on the PEP-C website under password protection. Grantee staff authorized to access the system will be able to download these cleaned analysis files in SAS or SPSS format.
The online data entry tools will reduce the grantees’ burden by facilitating the data entry process and minimizing coding and variable naming errors. The online data entry tools will also allow grantees without access to data management/analysis software to accurately quantify the information in their completed survey and dosage forms. They will be able to extract the data that they enter online in spreadsheet form, for their own local evaluation. This will reduce grantees’ evaluation burden by providing them with a means to code their completed questionnaires and dosage forms. PEP-C will conduct cross-site analyses to assess outcomes for the program as a whole.
The electronic multi-site data collection process will increase the efficiency and practical utility of the assessment of these programs. Similar online data collection tools have been developed and used by grantees in previous MAI cohorts and have been demonstrated to work efficiently and effectively. Based on feedback from previous cohorts of grantees, the questionnaires and procedures for electronic transmission of data files have been improved to increase efficiency and minimize burden. PEP-C’s Systems and MAI Cross-Site Teams will offer trainings to grantees and their project officers in collecting, storing, digitizing, and submitting their participant-level data. The PEP-C help-line will provide continuous technical assistance to grantees with specific questions about cross-site data processes.
SAMHSA/CSAP conducted an extensive literature search, consulted with staff in federal agencies and organizations that work with SA and HIV/AIDS prevention programs, and discussed the proposed program with substance abuse prevention experts. Specifically, SAMHSA/CSAP:
Conducted a comprehensive literature search of completed and ongoing studies of SA and HIV/AIDS prevention programs targeting youth and adults and found insignificant duplication with this cross-site evaluation. However, all reviewed studies were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or policy application of the results.
Consulted with staff in CSAT, CDC, NIAID, NIDA, ACF, OJJDP, HUD, DOE and DOJ. None of these Federal organizations has conducted a cross-site outcome evaluation of prevention and early intervention programs targeting minority youth and adults that is similar to the one being proposed for this evaluation.
Staff attended national meetings at which completed, ongoing, or planned evaluations were discussed and found insignificant duplication with the proposed evaluation.
Literature searches within SAMHSA’s three centers and with five other federal agencies were conducted to identify studies with similar goals and expected outcomes. These searches have indicated that no similar study has been conducted which examines prevention initiatives regarding substance abuse (SA) in relation to HIV/AIDS. Even though the present cross-site evaluation differs from others that have been conducted in the field, information generated from these literature searches has sharpened the present cross-site evaluation design and enhanced the likely utility of the results.
In summary, SAMHSA did not identify any redundancy in that there were no precedents for a cross-site evaluation of projects similar to the one being proposed. Related studies suffered methodological shortfalls such as low sample sizes, lack of dosage monitoring, and/or lack of published/reliable/valid measures and scales that make it unlikely that current information will be released among the scientific community or published in peer-reviewed journals, government publications, and other similar venues. Thus, it is clear that the data to be collected will be unique to the SAMHSA/CSAP MAI programs, collected only from participants of these programs, and are not available elsewhere. The data collected through this multi-site effort will be non-duplicative, minimize burden on respondents, and be of use to SAMHSA, minority serving institutions, and minority communities.
This data collection will have no significant impact on small entities.
Survey and dosage data will be collected from participants whose service duration is 30 days or longer at three points in time: baseline (program entry), program exit and three-to-six months post-exit; at two time points (baseline and exit) for services lasting between 2 and 29 days; and one time point (exit only) for services lasting a single day. The exit-only survey data collected from single-day participants will provide SAMHSA with accurate data on people served and their sociodemographic characteristics and will also be used to investigate whether the brief intervention made a significant impact on knowledge and/or attitudes, using an innovative analytic technique described in further detail in Section A.16 below.
Failure to collect the information from all participants at all indicated time points in time will result in missed opportunities to fully assess the impact of implemented programs in reducing risk, enhancing protection, and preventing problem behaviors associated with SA and HIV/HCV transmission. For example, baseline and exit data are minimal requirements for assessing program effects on participants. Data collected at all three points in time are essential, as they will enable an assessment of the sustainability of results after the program has ended, and if sustainable, for which types of interventions and populations. SAMHSA has taken into consideration the challenges that grantees encounter in tracking participants after the termination of their services, especially given the higher than average proportions of homeless and reentry populations served by the funded programs. The follow-up data collection requirement has therefore been limited to a subgroup of the participants, in order to reduce grantees burden.
The requirement to collect census data rather than sampling the participants is a response to the methodological needs of the cross-site evaluation. Since all participants do not enter the programs at the same time, but rather, start and end services in staggered fashion, it is not possible to draw up a complete sampling frame that will allow a random sample to be drawn. A sampling strategy that takes into account all selectivity factors would have to be customized at the level of the grant site, since a generic sampling design may not account for all site-specific patterns of outreach and intake. Having the grantees’ evaluators design a sampling strategy for their specific sites raises issues of comparability across sites. Additionally, the wide variability in the characteristics of grant sites, coupled with the nesting of participants within sites, calls for a multilevel analytic approach that requires sufficient power within as well as across grant sites. SAMHSA’s emphasis on reducing health disparities requires dependable evaluation results on the outcomes of vulnerable demographic and socioeconomic subgroups, further increasing the need for analytic power. These considerations necessitate a census, rather than a sample strategy for data collection.
Effective grant management and monitoring, continuous quality improvement of implemented prevention strategies, and future planning for prevention programs that address the behavioral health needs of at-risk minority populations all depend on the availability of dependable evaluation results.
The Individual Dosage Form will be completed by the provider for every one-on-one service encounter with each participant. Group Dosage Forms will be completed for every group session, and will list the unique identification numbers of all participants attending the session. The forms provide information about the types of service provided during the encounter and numbers of minutes of service duration for each type. This information will be linked to the participants’ survey records and will allow the cross-site team to analyze participant-level outcomes given types and dosage of services received. The results will provide information to policy makers, practitioners, and prevention researches about the types of services that provide the best results for specific population groups.
Without this information, SAMHSA will not be able to:
Determine the extent to which it can prevent, reduce, and/or delay substance abuse and, in turn, reduce other risky behaviors that can lead to HIV/AIDS infection among minority populations;
Monitor the quality of its prevention programs and determine how they can be improved to ensure continued success at meeting the needs of minority populations at risk for HIV/AIDS;
Describe fully the range of prevention services used and the efficacy of evidence-based programs;
Ascertain if participants are more knowledgeable about HIV/AIDS and how they relate to SA as a result of program participation;
Identify types and durations of prevention services that are most effective and identify the potentially unique behavioral health needs of minority populations;
Meet its federal reporting requirements to DHHS, OMB, and Congress.
This information collection fully complies with 5 CFR 1320.5(d) (2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on September 30, 2015 (80 FR 58742).
The multi-site evaluation and questionnaire design were based on initial consultation with SAMHSA experts from CMHS and CSAT, and on feedback from the previous HIV cohorts. Other SAMHSA HIV program experts, such as David R. Robertson of CSAT (among others), were consulted on the following issues:
Draft evaluation design and privacy/data security procedures; plan for coordinating and collecting data; measures to be used to assess outcomes; and mediating factors
Suitability of proposed assessment instruments
Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site evaluation and reporting the findings
Means of minimizing the burden on project staff and program participants
Identification of efforts to ensure user relevance of results.
CSAP consulted with other experts on SA and HIV/AIDS, as well as other Federal agencies with related programs or mandates, including NIDA, ACF, CDC, DOJ, OJJDP, HUD, and the DOE. Consultations resulted in the refinement of measures and the coordination of federal data needs.
Incentives (cash or in-kind) are not part of this cross-site data collection protocol.
SAMHSA has statutory authority to collect data under the Government Performance and Results Act (Public Law 1103(a), Title 31) and is subject to the Privacy Act for the protection of these data. As part of its grant application process, SAMHSA/CSAP requires grantees to describe the procedures they will use to ensure the privacy and protection of participant data. These include by whom and how the data will be collected; how data collection forms will be administered; where data will be stored; who will/will not have access to the information; and how the identity of participants will be safeguarded. The consent text at the beginning of the questionnaires includes language about privacy & anonymity and indicates that the data collection voluntary and will not affect participation in direct service programs. Data are collected through self-report and no identifying information such as name or Social Security number is available on completed questionnaires, dosage forms, or databases. Grantees are instructed to randomly assign a unique identification number to each participant, to identify data records with this number only, and to limit access to any master lists that link names with identification numbers. Grantees are also instructed to keep all completed questionnaires and dosage forms in a secure location separate from the location of the master lists and to limit access to authorized staff members.
Although demographic information is crucial for the evaluation and is included in the questionnaires, age is calculated from the year of birth only; month and day of birth are not included in the data to maximize privacy. In reporting outcomes by subgroup, no information is tabulated for subgroups with a sample size smaller than 20 individuals due to privacy considerations.
The information received from grantees and all other potential respondents will be kept private through all steps of the data processing, cleaning, analysis, and reporting process. PEP-C will password-protect and limit access to raw or cleaned MAI data; only team members directly responsible for the cross-site evaluation will be allowed access. All data analysts on the MAI cross-site team have received online training in federal data security standards and in conducting research on human subjects, provided by the National Institutes of Health. All data will be closely safeguarded, and no institutional or individual identifiers will be used in reports. Only aggregated data will be reported. SAMHSA and its contractors will not receive identifiable participant records. Provider-level information will be aggregated to, at minimum, the level of the grant/cooperative agreement-funding announcement (i.e. the cohort).
SAMHSA’s mission is to improve the quality and availability of prevention, early intervention, treatment, and rehabilitation services for substance abuse and mental illnesses, including co-occurring disorders, in order to improve health and reduce illness, death, disability, and cost to society. In carrying out this mission, it is necessary for service providers to collect sensitive information such as about the use of alcohol or drugs. The data that will be submitted by each grantee will be based in large part on data that most SA and HIV prevention programs routinely collect in order to understand the needs of program participants and to measure the impact of services. This primarily includes data on client demographics, substance abuse history, sexual practices, knowledge, beliefs, and attitudes toward SA and risky sexual behaviors, and services received. Grantees are required to have adequate consent procedures in place, and these procedures include obtaining and documenting active parental/guardian consent when necessary. SAMHSA review committees will not approve nor will SAMHSA fund a site without adequate provisions for meeting federal policies regarding consent and data privacy. SAMHSA grant review committees consider the following participant protection (PP) criteria:
Protect participants from potential risks,
Fair selection of participants,
Absence of coercion,
Valid and reliable data collection,
Privacy and confidentiality, and
Adequate consent procedures.
During the review process, committee members code grant applications as: “no PP concerns or comments;” or “PP comments or PP concerns.” Before an application can be funded, the applicant must address in writing any PP comments or concerns raised by the review committee. The SAMHSA participant protection officer then reviews these responses. Grantees are barred from funding until the participant protection office provides approval, especially if the application was coded with "concerns". After grantees respond to the protection officer’s satisfaction, the bar on funding is lifted.
SAMHSA follows procedures similar to those used by CDC and DOE regarding parental consent for youth; in addition grantees must adhere to state and local regulations.
Estimated Burden of the Youth and Adult Questionnaires
As mentioned earlier, the burden to participants of completing the questionnaires depends on the participant’s service duration, both in terms of the sections of the questionnaires to be completed and the number of time points at which data are collected. Exhibit 2 below displays the burden to participants per data collection time point, separately for the three service duration categories. The burden estimates for each service duration are based on the estimates of the instruments currently in the field (OMB # 0930-0298), adjusted for the reduction in the number of items as a result of the recent revision process.
Exhibit 2: Sections of the Questionnaires to be Completed and Total Burden to the Respondent per Data Collection Point, by Service Duration
Service Duration |
Sections of the Questionnaire To be Administered |
Number of Hours to Complete |
|
Youth |
Adult |
||
1 day or less |
Section One: Facts about You 3 to 5 questions from Section Two: Attitudes & Knowledge |
0.2167 |
0.2167 |
2 – 29 days |
Section One: Facts about You Section Two: Attitudes & Knowledge |
0.4333 |
0.3833 |
30 days or more |
Section One: Facts about You Section Two: Attitudes & Knowledge Section Three: Behavior & Relationships |
0.6167) |
0.5333 |
Analysis of the data submitted by Cohorts 7 through 10 indicates that not all participants with service duration longer than a single day are present at all of the required data collection points. Exhibit 3 displays the proportions responding to the survey at each required time point.
Exhibit 3: Response Rates of Participants with Service Duration Longer than a Single Day at Each Data Collection Point, by Service Duration
Service Duration |
Required Data Collection Points |
Youth Response Rates |
Adult Response Rates |
||||
Baseline |
Exit |
Follow-Up |
Baseline |
Exit |
Follow-Up |
||
2 – 29 days |
Baseline and Exit |
0.95 |
0.76 |
- |
0.97 |
0.76 |
- |
30 days or more |
Baseline, Exit, and 3 – 5 –Month Post-Exit Follow-up |
0.91 |
0.67 |
0.32 |
0.95 |
0.65 |
0.45 |
The following exhibit shows the expected number of data collection points per participant, adjusted for response rates. Please note that the available data do not contain information on the number of single-day participants who did not take the required exit survey. Lacking data on past response rates for this category of participants, the below exhibit makes the conservative assumption that all of them will take the required survey. This will lead to a slight over-estimate of the expected number of exit surveys taken by this group.
Exhibit 4: Required and Expected Numbers of Data Collection Points by Service Duration
Service Duration |
Required # of Data Collection Points |
Expected # of Data Collection Points (Exhibit 3) |
|
Youth |
Adult |
||
1 day or less |
1 |
1.00 |
1.00 |
2 – 29 days |
2 |
1.71 |
1.74 |
30 days or more |
3 |
1.90 |
2.05 |
Exhibits 5a and 5b are worksheets for calculating total annualized burden for the youth and adult questionnaires, respectively.
Exhibit 5a: Estimated Annualized Burden of the Youth Instrument
Service Duration |
Expected # of Respondents (Exhibit 1) |
Expected # of Responses per respondent (Exhibit 4) |
total # of responses |
# of Hours Per Response (Exhibit 2) |
Total Hour Burden |
Total Cost |
1 day or less |
64 |
1 |
64 |
0.2167 |
14 |
$161 |
2 – 29 days |
240 |
2 |
480 |
0.4333 |
208 |
$2,392 |
30 days or more |
1,136 |
2 |
2,272 |
0.6167 |
1,401 |
$16,112 |
Annualized Total |
|
|
|
|
1,623 |
$18,665 |
The total estimated burden of responding to the youth questionnaire is 1,623 hours per year. Assuming that the monetary value of an hour of the participant’s time is $11.50, this burden corresponds to $18,665 per year. The hourly cost estimate of $11.50 is based on the District of Columbia’s minimum wage planned to go into effect on 7/1/2016 (National Conference of State Legislatures, 2015). The minimum wage is an appropriate estimate of the monetary value of the respondents’ time, given that the respondents are below age 18 and the majority are full-time students.
Exhibit 5b: Estimated Annualized Burden of the Adult Instrument
Service Duration |
Expected # of Respondents (Exhibit 1) |
Expected # of Responses per respondent (Exhibit 4) |
total # of responses |
# of Hours Per Response (Exhibit 2) |
Total Hour Burden |
Estimated Cost* |
1 day or less |
1,040 |
1 |
1,040 |
0.2167 |
225 |
$2,588 |
2 – 29 days |
4,314 |
2 |
8,628 |
0.3833 |
3,307 |
$38,031 |
30 days or more |
19,150 |
2 |
38,300 |
0.5333 |
20,425 |
$234,888 |
Annualized Total |
|
|
|
|
23,957 |
$275,507 |
* Estimated at $11.50 per hour.
The total estimated burden of responding to the adult questionnaire is 23,957 hours per year. Assuming that the monetary value of an hour of the adult participant’s time is $11.50, this burden corresponds to $275,507 per year. The minimum wage is an appropriate estimate of the monetary value of the respondents’ time, given the target population of mostly college students and high-risk young adults. The justification for using this particular figure is that at the time of this writing, it was the highest state-mandated minimum wage in the nation for which the exact value was known.
Estimated Burden of the Individual and Group Dosage Forms
An Individual Dosage Form will be completed for each one-on-one service encounter with a participant, and takes, on average, three minutes (0.0500 hours) to complete. The number of service encounters that a participant has depends on the total duration of his/her services. Exhibit 6 displays the worksheet for estimating the total annualized burden of the Individual Dosage Form.
Exhibit 6: Estimated Annualized Burden of the Individual Dosage Form
Service Duration |
Expected # of Participants (Exhibit 1) |
Expected # of Service Encounters per Participant |
total # of Forms Completed |
Total Hour Burden |
1 day or less |
1,104 |
1 |
1,104 |
55 |
2 – 29 days |
4,554 |
4 |
18,216 |
911 |
30 days or more |
20,286 |
8 |
162,288 |
8,114 |
Annualized Total |
25,944 |
|
181,608 |
9,080 |
The total annualized burden of the Individual Dosage Form is 9,080 hours per year.
A Group Dosage Form will be completed for each group session held by the funded programs. The only difference between Individual and Group Dosage Forms is that participant identification numbers of the multiple attendees need to be entered for each group session. Further, the Group Dosage Form has space for entering up to three service codes and durations as compared to four in the Individual Dosage Form. Although the form has room for entering up to 80 attendee numbers, the typical group session has approximately 20 attendees. Entering this many participant numbers takes approximately five minutes, bringing the burden to eight minutes (0.1333 hours) per form. Each grantee holds approximately 26 group sessions per year; spending a total of three hours completing Group Dosage Forms. The total annualized burden of the Group Dosage Form across all 138 grantees is, therefore, 478 hours.
It is worth noting here that a typical program participant receives a combination of group and individual services and will have multiple service encounters of both types. Additionally, many group-format interventions take multiple sessions, each of which will require a separate Group Dosage Form. This implies that the above estimates of the annual numbers of completed forms cannot be used to deduce the number of participants served each year.
Given that the dosage forms are completed by grantee staff, most of whom are community and social service workers, it is appropriate to estimate the monetary value of their hour burden as the mean hourly wage for this occupational group. According to the Bureau of Labor Statistics (2015), this figure was $21.79 in May, 2014. Applying this wage rate to the estimated hour burden, the annualized costs of the Individual and Group Dosage Forms are estimated to be $197,862 and $10,422, respectively.
Exhibit 7 summarizes estimates of the total annualized burden for the four instruments
Exhibit 7a: Total Estimated Annualized Burden by Instrument
Type of respondent activity |
Number of Respondents |
Responses per Respondent* |
Total Responses |
Hours per Response |
Total Burden Hours |
Wage Rate |
Total Hour Cost |
Youth Questionnaire/ Single-day service duration |
64 |
1 |
64 |
0.2167 |
14 |
$11.50 |
$161 |
Youth Questionnaire/ 2-29-day service duration |
240 |
2 |
480 |
0.4333 |
208 |
$11.50 |
$2,392 |
Youth Questionnaire/ 30-or-more-day service duration |
1,136 |
2 |
2,158 |
0.6167 |
1,401 |
$11.50 |
$16,112 |
Adult Questionnaire/ Single-day service duration |
1,040 |
1 |
1,040 |
0.2167 |
225 |
$11.50 |
$2,588 |
Adult Questionnaire/ 2-29-day service duration |
4,314 |
2 |
8,628 |
0.3833 |
3,307 |
$11.50 |
$38,031 |
Adult Questionnaire/ 30-or-more-day service duration |
19,150 |
2 |
38,300 |
0.5333 |
20,425 |
$11.50 |
$234,888 |
Individual Dosage Form |
138 |
1,316 |
181,608 |
0.0500 |
9,080 |
$21.79 |
$197,862 |
Group Dosage Form |
138 |
26 |
3,588 |
0.1333 |
478 |
$21.79 |
$10,422 |
Total |
26,220 |
|
235,980 |
|
35,139 |
|
$502,456 |
*Adjusted for expected response rates
The estimated total annualized burden of the four instruments for which approval is being requested is 35,139 with a total cost burden to participants and program staff of $502,456.
Exhibit 7b: Summary Total Estimated Annualized Burden
Type of respondent activity |
Number of Respondents |
Responses per Respondent* |
Total Responses |
Hours per Response |
Total Burden Hours |
Youth Questionnaire |
1,440 |
1.98 |
2,851 |
.4222 |
|
Adult Questionnaire |
24,504 |
1.956 |
47,933 |
.508 |
24,378 |
Individual Dosage Form |
138 |
1,316 |
181,608 |
0.0500 |
9,080 |
Group Dosage Form |
138 |
26 |
3,588 |
0.1333 |
478 |
Total |
26,220 |
|
235,980 |
|
35,139 |
There will be no capital, start up, or operation and maintenance costs.
The contract award to the PEP-C’s MAI Cross-Site Evaluation Task will cover all aspects of the evaluation design, planning, data collection, processing, and analysis, with the annualized cost of $530,000. These costs cover the following activities: assistance to grantee sites in cooperation with the national evaluation; cleaning and processing of outcome data from grantee sites; data analysis and reporting; and development of public use data and documentation.
It is anticipated that the Government Project Officers (PO) who oversee the grants will expend time working with the PEP-C team and grantees in fulfilling the requirements of the program associated with the collection and submission of data. The team lead of the MAI POs and the director of CSAP’s Division of Community Programs (CSAP-DCP) will attend weekly meetings with the PEP-C team and will review and approve all cross-site materials, including reports, conference presentations, and training materials. The Contract Officer’s Representative (COR) and the alternate COR (ACOR) overseeing the PEP-C contract will expend a portion of time overseeing all aspects of the MAI cross-site evaluation.
Exhibit 8 displays the estimation worksheet for the annualized total cost to the government. Annual hours are based on a 40-hour work week for 48 weeks per year, that is, 1,920 hours per annum. It is estimated that 8 CSAP Project Officers will be involved in monitoring the grants. Total cost to the government is estimated at $606,800.
Exhibit 8: Estimated Annualized Cost to the Government
Position |
Percent FTE |
Annual Hours |
Rate |
Total Annual Cost |
PEP-C Contract |
NA |
NA |
NA |
$530,000 |
PEP-C COR |
15% |
288 |
$60 |
$17,280 |
PEP-C ACOR |
5% |
96 |
$60 |
$5,760 |
CSAP-DCP Director |
5% |
96 |
$60 |
$5,760 |
MAI Project Officer Team Lead |
5% |
96 |
$60 |
$5,760 |
MAI Project Officers (Based on Estimated 8 POs) |
5% |
768 |
$55 |
$42,240 |
Total |
|
|
|
$606,800 |
Currently there were 16,770 hours in the OMB inventory. SAMHSA is requesting 35,139 to continue this cross-site evaluation. The additional 18,369 hours are due to program changes, including an increase in the number of grantees funded by the MSI CBO and CBI programs and related increase in the expected numbers of participants to be served. Additionally, the estimates of the number of participants by service duration and of response rates to the surveys are based on data submitted by more recent cohorts and are more representative of the probable activities of the current cohorts of grantees. Finally, two brief dosage forms have been added to the data collection protocol.
Analysis Plans
The defining characteristic of this cross-site evaluation is the sharing of a common data collection protocol, a common set of performance measures, outcome objectives, and evaluation questions by all participating grantees. This evaluation differs from more traditional multi-site clinical trials because each individual grantee will select Evidence Based Interventions (EBI) and Diffusion of Effective Behavioral Interventions (DEBI) that are adapted to the needs of the particular target population, setting, and organizational structure. This multi-site evaluation does not test a single intervention that has different settings; rather, it tests intervention profiles that have similar outcome objectives but that use different approaches to accomplish those objectives.
Analysis of a multi-site data set requires a complex set of interrelated tasks. Planning for these tasks must be flexible and allow adjustments as the opportunities and challenges presented by the empirical realities of the data set are encountered and analysis plans adjusted accordingly. While multi-site studies provide strong opportunities for knowledge generation (because of the ability to compare and contrast intervention and implementation variation in a single evaluation), they also present significant evaluation challenges. This evaluation recognizes those challenges and anticipates solutions as they will apply to the 138 participating grantees.
Sample Size Determination. Individual grantees have proposed their target population sizes in their strategic prevention plans. The establishment of sample size at the grantee level depends to some extent on financial constraints for program intervention services, staff allocation, outreach and participant retention activities, and evaluation activities, including stipends. Analysis of data submitted by cohorts 7 through 10 indicate that on average, a single grant site collects data from approximately 185 participants per year. This means that the 84 MSI CBO grantees will submit data for 555 participants per grantee during their three-year grant period, for a total sample size of 46,620. The 54 CBI grantees will submit data for 925 participants per grantee during their five-year grant period, for a total sample size of 49,950. The entire sample size for the final cross-site report is, estimated to be 96,570. However, it should be noted that not all participants complete all sections of the survey. Therefore analyses of individual measures will be based on smaller numbers of data records. Analyses of attitudes and knowledge, included in the questionnaires of all participants with service duration longer than a single day, are expected to have larger sample sizes than those of behavioral outcomes, only included in the questionnaires of participants whose service duration is 30 days or longer.
Statistical Procedure Determination
As a multi-site design, this initiative collects information at two levels of observation: 1) participant-level and 2) grantee level. At the participant level, each individual’s survey records at multiple time points and dosage data from each service encounter will be linked to create a single data record containing all demographic, outcome, service type, and dosage information for each participant. At the level of the grant site, information such as the type of community, grantee organizational structure, types of interventions implemented, infrastructure building efforts, and grant funding allocated to various types of activities are available from grantees’ quarterly progress reports for which OMB approval is being requested through a separate statement. For analysis purposes, relevant grantee-level characteristics will be linked to the participant-level database to produce a hierarchical structure with participants nested within grant sites (Straw & Herrell, 2002).
The proposed analysis includes several distinct steps:
First, pooled analyses of outcomes will be conducted to assess overall program effects and their sustainability. Program effects will be evaluated through paired comparisons of baseline and exit values. Sustainability of effects will be evaluated through paired comparisons of baseline and follow-up values. Past analyses have suggested that some measures continue to improve after program exit (see Table 3). To assess this post-exit improvement, paired comparisons between exit and follow-up values will also be conducted.
Second, hierarchical linear modeling will be used to measure program effects while accounting for the interclass correlations among participants of the same grantee. These models will be expanded to identify and control for moderating factors at both the participant and grantee levels. This multivariate analysis will yield information about variations in outcomes due to the sociodemographic characteristics and baseline risk levels of individual participants and due to differences in grantee characteristics.
The data collection protocol for participants who stay in the program for a single-day requires administration of the questionnaire only at exit. Outcome analysis of these data will compare the exit responses of each participant with a norm constructed using the baseline responses to the item provided by a comparable group of participants from the same site who were in longer interventions (and hence, took both the baseline and exit surveys). This approach will provide an approximate assessment of the effect of the single-day intervention on the participants’ relevant knowledge and attitudes. . Given that grantees have a choice in the 3-to-5 attitude/knowledge items they include in the single-day exit survey, the sample size for each outcome measure will be relatively low. This third step will only be conducted after the number of available single-day survey records becomes large enough to provide sufficient power for the analysis.
Analysis Techniques and Statistical Test Determination
In assessing overall improvement based on pooled data, paired comparison tests appropriate to the level of measurement will be employed. For dichotomous outcome measures, significance will be tested using McNemar’s test (Lidell, 1976; Yang, Sun, & Hardin, 2010). For normally-distributed continuous outcome measures, matched-pairs t-tests will be used to assess significance. The significance of change in ordinal or skewed continuous measures will be tested using the Wilcoxon signed-rank test (Wilcoxon, 1945; Blair & Higgins, 1980). Finally, single-sample t-tests will be conducted to test the significance of program effects based on the exit-only data from single-day-service participants.
A multi-level analysis approach [e.g., HLM] will be used to investigate the effects of participant- and grantee-level characteristics on participant outcomes. Characteristics hypothesized to have a bearing on program effects, such as choice of prevention strategy and type of grantee organization, will be included in the dataset together with participant-level baseline, exit, and follow-up survey records and dosage data. Nesting participant-level data within program-level data in this fashion will allow the construction of multi-level causal models that simultaneously test for the effects of participant and program characteristics on program outcomes and to identify significant interactions between these two levels, while accounting for the nested data structure (Osborne, 2000; Raudenbush & Bryk, 2002).
A two-step analysis strategy will be used to control for attrition bias. First, baseline characteristics of all participants will be compared with baseline characteristics of participants who completed the programs. These two sets of data records will be compared with respect to baseline values of demographic characteristics, levels of risk and protection, and levels of substance use. Second, baseline factors found to differ significantly between the two sets of records will be included in all of the models and only model estimates net of these factors will be reported in the final evaluation results. This approach minimizes selectivity biases due to program attrition in reported program effects.
Given that target populations and community contexts vary by grant site, the multi-site dataset will contain data from a wide range of program participants in terms of their demographic, socioeconomic, and cultural characteristics. All multivariate models will include all of the factors that account for the differences among groups. Those factors that are found to have a significant effect on outcomes will be identified and interaction terms will be constructed to represent differences in program effects due to recruitment strategy. This analytic strategy will allow the evaluation to take into consideration the mediating effects of a broad range of factors on program outcomes. The inclusion of these demographic, socioeconomic, and cultural control variables and interaction terms in the models will also ensure that final results are not biased toward the outcomes of groups with relatively large numbers of data records.
Reporting and Dissemination Plan
In addition to the annual cross-site reports and presentations of results to SAMHSA project officers and grantees, the MAI cross-site evaluation results will be made available to the public through publications and conference presentations. The following journals carry articles on SA prevention and HIV/AIDS and are expected to serve as potential vehicles for distribution of evaluation results: Journal of Substance Abuse Treatment, International Journal of Addictions, Journal of Community Psychology, Journal of Adolescent Research, Journal of Adolescent Health, Preventive Medicine, Evaluation Review, Policy Studies Review, and The American Journal of Public Health. Evaluation results could also be published in other journals that focus on HIV/AIDS. These include The Journal of the American Sexually Transmitted Disease Association, Health Education and Behavior, AIDS: Official Journal of the International AIDS Association, AIDS Education and Prevention, The Journal of Sex Research, AIDS Care, Psychological and Socio-Medical Aspects of AIDS/HIV, and Current Opinion in HIV and AIDS. Evaluation results are also targeted for publication in journals focusing on infectious diseases. These include, among others, The Journal of the American Microbiological Association and Journal of Infectious Diseases.
The evaluation results will be distributed through presentations at annual conferences of national and international public health, prevention, and program evaluation organizations, such as the Society for Prevention Research, the American Public Health Association, the National Association of Alcohol and Drug Abuse Counselors, The National Prevention Network, the American Evaluation Association, The Society for Prevention Research, and HIV/AIDS national meetings as well as regional and State SA prevention and treatment associations. HIV/AIDS meetings could include, among others, CDC Annual Conferences on AIDS and Conferences of the International AIDS Society. Results could also be presented at meetings focusing on infectious diseases such as annual meetings of the American Society of Microbiology.
Documents will also be prepared and published on behalf of the government (CSAP) through the Government Printing Office (GPO) for Federal agency and public use. Findings will also be available via OMB’s Website: www.expectmore.gov, as well as in annual reports to Congress and the performance detail sections of annual SAMHSA budgets as they become publicly available.
Timeline
MAI’s MSI CBO grants are three-year programs while the CBO grants are funded up to five years. Typically, there is an initial period devoted to Steps 1, 2 and 3 of the SPF, namely conducting needs assessment, capacity building, and strategic planning. Grantees begin implementation and data collection only after their proposed strategic prevention plans are approved by their SAMHSA project officers. The approximate dates of the SPF milestones for the four cohorts in this cross-site evaluation are presented in Exhibit 9 below.
Exhibit 9: Project Timelines
|
MSI CBO 2013 (29 3-year grants) |
MSI CBO 2014 (21 3-year grants) |
MSI CBO 2015 (34 3-year grants) |
CBI 2015 (54 5-year grants) |
Needs Assessment, Capacity Building, Strategic Planning |
10/2013 – 6/2014 |
10/2014 – 6/2015 |
10/2015 – 6/2016 |
10/2015 – 6/2016 |
Implementation, Data Collection |
6/2014 – 9/2016 |
6/2015 – 9/2017 |
6/2016 – 9/2018 |
6/2016 – 9/2020 |
Follow-up, Evaluation, and Reporting |
Ends 12/31/2016 |
Ends 12/31/2017 |
Ends 12/31/2018 |
Ends 12/31/2020 |
Please note that the annualized burden estimates presented in previous sections are for a typical year during which the grantee is implementing interventions. There is no burden factored in for the period spent on the first three stages of the SPF. It should also be kept in mind that the Capacity Building Initiative (CBI) grantees are expected to spend a longer time implementing the pre-implementation steps of the SPF, given that the initiative has higher emphasis on building a prevention infrastructure to meet the specific needs of the local community.
The expiration date will be displayed.
The certifications are included in this submission.
REFERENCES
Alter, M.J. (2002). Prevention of spread of hepatitis C. Hepatology, Supplement 5B: Management of Hepatitis C, 36, s93–s98.
Bellamy, N.D., Wang, M.Q., Matthew, R.F., Leitao, M.P., Agee, R.A., & Yan, A.F. (2008). Structural model analysis of HIV risk behaviors among sexually active minority adolescents. Journal of the National Medical Association, 100, 914 – 924.
Blair, R. C. & Higgins, J.J. (1980). A Comparison of the power of Wilcoxon's rank-sum statistic to that of Student's t statistic under various nonnormal distributions. Journal of Educational Statistics, 1, 309–335.
Brodbeck J., Matter M., & Moggi F. (2006). Association between cannabis use and sexual risk behavior among young heterosexual adults. AIDS Behavior, 10, 599 – 605.
Bureau of Labor Statistics. (2015). Occupational Employment and Wages, 2014. United States Department of Labor. Accessed on July 20, 2015 from http://www.bls.gov/oes/current/oes210000.htm.
Centers for Disease Control and Prevention. (2015a). HIV/AIDS Surveillance Slide Sets. National Center for HIV/AIDS, Viral Hepatitis, STD & TB Prevention, Division of HIV/AIDS Prevention. Accessed on June 30, 2015 from http://www.cdc.gov/hiv/library/slideSets/index.html.
Centers for Disease Control and Prevention. (2015b). Viral Hepatitis Surveillance United States, 2013. Accessed on July 2, 2015 from http://www.cdc.gov/hepatitis/statistics/2013surveillance/pdfs/2013hepsurveillancerpt.pdf.
Fisher, J.C., Bang, H., & Kapiga, S.H. (2007). The association between HIV infection and alcohol use: A systematic review and meta-analysis of African studies. Sexually Transmitted Diseases, 34, 856 – 863.
Garfein, R.S., Vlahov, D., Galai, N., Doherty, M.C., & Nelson, K.E. (1996). Viral infections in short-term injection drug users: the prevalence of the hepatitis C, hepatitis B, human immunodeficiency, and human T-lymphotropic viruses. American Journal of Public Health, 86, 655-661.
Institute of Medicine. (2010). Hepatitis and liver cancer: a national strategy for prevention and control of hepatitis B and C. Washington, DC: The National Academies Press.
Liddell, D. (1976). Practical tests of 2 × 2 contingency tables. Journal of the Royal Statistical Society, 25, 295–304.
National Conference of State Legislatures. (2015). 2015 Minimum wages by state. Accessed on July 5, 2015 from http://www.ncsl.org/research/labor-and-employment/state-minimum-wage-chart.aspx.
Osborne, J. W. (2000). Advantages of hierarchical linear modeling. Practical Assessment, Research & Evaluation, 7. Accesed on July 12, 2015 from http://PAREonline.net/getvn.asp?v=7&n=1.
Raudenbush, S. W. & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd Edition). Thousand Oaks, CA: Sage Publications.
Rondinelli, A.J., Ouellet, L.J., Strathdee, S.A., Latka, M.H., Hudson, S.M., Hagan, H., & Garfein, R.S. (2009). Young adult injection drug users in the United States continue to practice HIV risk behaviors. Drug and Alcohol Dependence, 104, 167-74.
Straw, R.B. & Herrell, J. M. (2002). Conducting multiple site evaluations in real-world settings. San Francisco, CA: Jossey-Bass.
Terrault , N.A. Sexual activity as a risk factor for hepatitis C. (2002). Hepatology, Supplement 5B: Management of Hepatitis C, 36, s99–s105.
Wang, M.Q., Matthew, R.F., Chiu, Y.-W., Yan, F., & Bellamy, N.D. (2007). Latent model analysis of substance use and HIV risk behaviors among high-risk minority adults. Journal of Alcohol & Drug Education, 51, 35 – 64.
Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin 1, 80–83.
Yang, Z., Sun, X., Hardin, J.W. (2010). A note on the tests for clustered matched-pair binary data". Biometrical Journal, 52, 638–652
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | HIV OMB Supporting Statement |
| Author | proddy |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |