Quarterly Progress Reporting and Indirect Services Outcome Data Collection for the Minority AIDS Initiative (MAI) Substance Abuse and HIV Prevention Programs
OMB Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) is requesting approval from the Office of Management and Budget (OMB) for the collection of quarterly progress information and annual community-level outcome data from CSAP’s Minority AIDS Initiative (MAI) programs. The two new instruments supported by the present statement are:
Quarterly Progress Report
Indirect Services Outcomes (ISO)
HIV Testing Retrospective Reporting Tool (to be completed once only)
The 158 MAI grantees that will use these instruments are community-based organizations (CBOs) and minority serving educational institutions (MSIs) that serve high-risk minority populations with the goal of preventing substance abuse and HIV & viral hepatitis (VH) transmission through integrated prevention services and referrals to treatment as needed. The Quarterly Progress Report is intended to standardize and facilitate quarterly progress reporting required from all grantees as part of their grant agreements. By providing a standard online instrument for collecting information on grantees’ quarterly activities, accomplishments, and challenges, SAMHSA aims to reduce the burden of this quarterly reporting requirement on the grantees, assist the government project officers (PO) in monitoring the grantees’ performance, and help identify any technical assistance needs in a timely fashion. The Quarterly Progress Report will also meet new demands for data from various sources including Congress, Department of Health and Human Services (HHS), Government Accountability Office (GAO), and several other Federal agencies. In particular, HHS has asked Federal agencies that provide funding for HIV prevention and treatment programs, to collect seven core HIV measures. SAMHSA/CSAP has agreed to collect two of these measures: (i) Number of individuals tested for HIV & those who tested positive, and (ii) Number of homeless individuals tested for HIV. The Quarterly Progress Report includes fields for grantees to enter aggregate data on the number of individuals tested for HIV, their distribution across demographic categories and housing status, numbers tested for the first time, and numbers with HIV-positive results. In light of the increasing emphasis on the prevention of viral hepatitis, the new grantees are being encouraged to provide testing and vaccination services for Hepatitis C as well. The Quarterly Progress Report therefore includes data fields for reporting aggregate data on viral hepatitis testing and vaccination activities.
All CSAP grantees are required to organize their prevention activities around the Strategic Prevention Framework (SPF). This framework consists of five interrelated steps and two overarching principles. The first step for all grantees is to conduct a needs assessment in their target communities and to submit a report summarizing their findings. Grantees next work on building their prevention capacity to meet the needs identified in the previous step. The third step is strategic planning which includes identifying targets, selecting effective community prevention programs, policies, and practices that best align with the needs of the community. The strategic plan developed during this phase is submitted to CSAP and reviewed by the grantee’s PO. The plan is revised in line with the PO’s feedback and once approved by the PO, it is put into action, initiating the fourth SPF step, that is, implementation. The final step is evaluation of outcomes. All grantees are required to submit an evaluation report at the end of their grant period. The two guiding principles of the SPF are sustainability and cultural competence. Grantees are encouraged to direct their efforts toward infrastructure building and program implementation strategies that are likely to sustain their effects after the end of the grant. Additionally, all activities have to be planned and executed with careful consideration of the specific cultural and linguistic needs of the targeted groups.
Given that the success of each SPF step is dependent on the competence with which the previous step was executed, there is need for constant monitoring of grantees’ progress through the process. This is especially true of grantees that are new to the SPF, but all grantees moving through the SPF steps need to be closely monitored, their barriers promptly addressed, and their training and technical assistance needs met without delay. The overall purpose of the Quarterly Progress Report is to facilitate communication between grantees and their POs about the progress of the grantee through the steps of the SPF. Although POs also communicate with their grantees through regular conference calls and site visits, the Quarterly Progress Report provides them with a standard tool for assessing their grantees’ progress. It also regulates the exchange of information about the grantees’ accomplishments and barriers.
The ISO is a brief instrument for collecting annual data on community-level measures related to the indirect-service interventions (environmental strategies and information dissemination) implemented by each grantee. The grantees that will be using these instruments are all required to include such interventions in their strategic prevention plans in an effort to garner observable changes at the community level. Past evaluations of SAMHSA’s programs and other studies of community prevention have shown that implementing community-wide, population-based strategies (e.g. social norms campaigns, social marketing, changes in policies, regulations, and systems), combined with direct-service interventions delivered directly to program participants (e.g., health education, counseling, testing, problem identification and referral) have broad and lasting desirable impacts on risk and protective factors associated with behavioral health (Foxcroft & Tsertsvadze, 2011); Willard, Chutuape, Stines, & Ellen, 2012). The ISO will allow CSAP to assess the effectiveness of grantees’ indirect-service interventions in achieving community-wide change, as mandated by the terms of their grants.
The third data collection instrument for which approval is being sought is intended to collect FY 2015 data on the HIV testing activities of the grantees. It will be used once only, immediately after the system goes online, in order to collect data for two of the seven HHS Core Indicators that SAMHSA/CSAP has agreed to report. Although this statement refers to it as a separate instrument for purposes of clarity in burden estimation, it has the same data fields as the HIV Testing Implementation section of the main Quarterly Progress Report tool and differs only in its reporting timeframe.
In addition to reducing grantees’ quarterly reporting burden and assisting POs in their grant management activities, the Quarterly Progress Report and ISO will provide valuable data for the MAI cross-site evaluation, currently conducted by CSAP’s Program Evaluation for Prevention Contract (PEP-C). Standardized and quantifiable data about the grantees’ planning and implementation data, collected with the Quarterly Progress Report, will be merged with participant-level data collected through participant-level instruments for which approval is being sought through a separate submission (see Federal Register, Vol. 80, No. 189, Wednesday, September 30, 2015, pp. 58742 – 58744), to address evaluation questions about the types of strategies or combination thereof that produce the most favorable outcomes for individual participants. The community-level outcome data collected through the ISO will also be merged into this multilevel database and will be analyzed in conjunction with the individual-level outcomes.
This data collection effort directly supports two of SAMHSA’s six Strategic Initiatives, namely Prevention of Substance Abuse and Mental Illness, and Health Care and Health Systems Integration while also addressing the Healthy People 2020 Substance Abuse Topic Area HP 2020-SA. The primary objectives of the effort are to:
Ensure the correct implementation of the five SPF steps by maintaining a continuous feedback loop between grantees and their POs;
Promptly respond to grantees’ needs for training and technical assistance;
Assess the fidelity with which the SPF is implemented;
Collect aggregate data on HIV testing to fulfill SAMHSA’s reporting and accountability obligations as defined by the Government Performance and Results Modernization Act (GPRA Modernization Act) and HHS’s HIV Core Measures;
Assess the success of the MAI in reducing risk factors and increasing protective factors associated with the transmission of the Human Immunodeficiency Virus (HIV), Hepatitis C Virus (VH) and other sexually-transmitted diseases (STD);
Measure the effectiveness of evidence-based programs and infrastructure development activities such as: outreach and training, mobilization of key stakeholders, substance abuse and HIV/AIDS counseling and education, testing, referrals to appropriate medical treatment, and other intervention strategies (e.g., cultural enrichment activities, educational and vocational resources, motivational interviewing & brief interventions, social marketing, and computer-based curricula);
Investigate intervention types and features that produce the best outcomes for specific population groups;
Assess the extent to which access to health care was enhanced for population groups and individuals vulnerable to behavioral health disparities residing in communities targeted by funded interventions, thereby reducing disparities in provision of health care for these vulnerable groups.
These objectives support the four primary goals of the National HIV/AIDS Strategy which are: 1) reducing new HIV infections, 2) increasing access to care and improving health outcomes for people living with HIV/AIDS, 3) reducing HIV-related disparities and health inequities, and 4) achieving a coordinated national response to the HIV epidemic.
This program is authorized by Section 516 of the Public Health Service Act, as amended, and subject to the availability of funds. It was supported by the Congressional Black Caucus through its Conference Report on H.R. 4328, Making Omnibus Consolidated and Emergency Supplemental Appropriations Act, for FY 1999 (House of Representatives, October 19, 1998), to address prevention and treatment needs of minority communities that are disproportionately affected by HIV/AIDS. It builds on previously authorized programs addressing these issues (discussed below).
Although several Federal agencies have mandates to fund projects targeting minority populations who are at risk for substance abuse and HIV/AIDS, our knowledge of the efficacy of such programs once they become widely disseminated is still limited. Prior efforts to evaluate federal substance use prevention initiatives targeting at-risk populations have focused on highly specific program models and narrowly defined target populations or have been hampered by lack of valid instrumentation and poor study design. Although models have been disseminated to community-based agencies (that typically implement these programs under less rigorous and controlled parameters), measures and efforts to assess outcomes were inadequate and/or not sufficiently designed to determine the full impact of these interventions, especially on vulnerable populations. In addition, the link between substance abuse and HIV/AIDS outcomes has not been sufficiently evaluated for these programs or in local community settings.
The data collection results will have significant implications for the substance abuse and HIV/AIDS prevention field, the allocation of grant funds, and other evaluation activities conducted by multiple Federal, State, and local government agencies. The results will be used to develop Federal policy in support of SAMHSA program initiatives, inform the public of lessons learned and evaluation findings, improve existing programs and promote replication and dissemination of effective prevention strategies.
Background
Epidemiological studies on the dynamics of substance abuse and HIV/AIDS demonstrate a continued need to reach out to communities of color, particularly to those reporting high rates of HIV/AIDS and other sexually transmitted diseases (STDs). According to 2013 surveillance data from the U.S. and its six territories reported by the Centers for Disease Control and Prevention (CDC, 2015a), the rate (per 100,000) of HIV infection was 105.7 among Black/African Americans and 41.8 among Hispanic/Latinos, but only 13.8 among Whites.
Of particular concern to communities of color is the high level of HIV transmission among young people, particularly young Black/African American men who have sex with men (MSM). According to CDC surveillance data, there were 40,634 individuals between the ages of 13 and 24 living with an HIV infection in the U.S. and its territories, as of the end of 2012. Of these adolescents and young adults, less than 15% identified themselves as White; the rest belonged to communities of color.
There are multiple psychosocial factors that render specific minority populations especially vulnerable to Substance Abuse (SA) and HIV transmission, such as stigma, homophobia (experienced by gay, lesbian, bisexual, or transgender individuals), poverty, lack of health insurance, and lack of access to high-quality prevention and treatment services. Regardless of the mode of transmission, HIV/AIDS is an infectious disease that has drastic long-term medical, economic and social consequences on minority populations. Meeting the challenges posed by HIV/AIDS requires close coordination with existing local, State, and territorial substance abuse and HIV/AIDS prevention programs.
According to the Institute of Medicine (IOM), VH is another growing concern in the United States (IOM, 2010). Cases of acute VH have recently increased from 781 in 2009 to 2,138 in 2013. During the same period, the incidence rate more than doubled, increasing from 0.3 to 0.7 per 100,000. In 2013, individuals between the ages of 20 and 29 had the highest incidence rate with 2.01 acute cases per 100,000. When broken down by race and ethnicity, the rate in 2013 was highest among American Indians/Alaska Natives (1.7), followed by White non-Hispanics (0.82). The rates among Black non-Hispanics and Hispanics were 0.20 and 0.22, respectively (CDC, 2015b). In response to the recent emergence of VH as a national public health concern, SAMHSA has included VH testing, referral, and prevention efforts among the activities required of the recent-awarded MAI grantees, starting with grantees funded in 2014.
HIV and VH share several risk factors, such as unprotected sexual intercourse and injection drug use (Garfein, Vlahov, Galai, Doherty, & Nelson, 1996; Terrault, 2002; Rodinelli et al., 2009). In addition to exacerbating the symptoms of viral hepatitis through its adverse effects on liver functions, alcohol abuse also poses a risk for both HIV and VH by increasing the likelihood of risky sexual behaviors and of noncompliance with treatment plans (Alter, 2002; Fisher, Bang, & Kapiga, 2007). Likewise, marijuana use has been shown to increase the incidence of risky sexual behaviors, especially among young adults (Brodbeck, Matter, & Moggi, 2006). These findings indicate a national need for integrated SA, HIV, and VH prevention services. SAMHSA has responded to this need by structuring the requests for applications for MAI grants to require an integrated approach that combines SA, HIV, and VH prevention interventions with HIV and VH testing, counseling, and referrals for treatment. The MAI is working to improve access to quality services by increasing outreach and service capacity to at risk populations of color. By applying the SPF model to develop and implement integrated strategies for the prevention of both substance abuse and HIV/AIDS transmission, the initiative is making an important contribution to the national effort to meet these public health needs.
MAI Grantees
There will be 158 grantees funded through two MAI programs with slightly different emphases implementing the data collection protocol for which approval is being requested. MAI’s Minority Serving Institutions (MSI) in Partnerships with Community-Based Organizations (CBO) Program provides grants to Historically Black Colleges and Universities, Hispanic Serving Institutions, American Pacific Islander Serving Institutions, and Tribal Colleges and Universities in partnership with CBOs in their surrounding communities to provide integrated SA, HIV, and VH prevention services to young adults. The second, MAI’s Capacity Building Initiative (CBI) funds community-level domestic, public and private nonprofit entities, federally recognized American Indian/Alaska Native Tribes and tribal organizations, and urban Indian organizations. CBI grants focus on building a solid infrastructure for integrated SA, HIV, and VH prevention service provision and implementation of evidence-based prevention interventions. The target population for the CBI grantees is at-risk minority adolescents and young adults. The Quarterly Progress Report and ISO instruments will be completed by authorized staff at these 158 grantee organizations.
The purpose of this data collection is to inform program direction and identify and address program weaknesses. SAMHSA/CSAP must also collect these data to meet its federal requirements specified in the Government Performance and Results Act (GPRA) of 1993 and the GPRA Modernization Act of 2010 (PL 111-352). The information collected through the Quarterly Progress Report and ISO instruments will also be used for various federal government reports including:
Congressional HIV Testing Report (annual)
National HIV/AIDS Strategy progress report (annual)
Viral Hepatitis Action Plan (VHAP) progress report (annual)
Office of HIV/AIDS and Infectious Disease Policy (OHAIDP) African American Inventory report (annual)
White House Minority AIDS Initiative progress report (annual)
White House Minority Serving Institutions progress report (annual)
Secretary’s Minority AIDS Initiative Fund (SMAIF) progress report (annual for ongoing SMAIF-funded programs – CSAP is completing one SMAIF program in 2015)
Office of National AIDS Policy (ONAP) implementation updates (semi-annual)
In addition, SAMHSA provides data for various Office of HIV/AIDS and Infectious Disease Policy (OHAIDP) reports and numerous ad hoc reports, as well as reports issued by the Government Accounting Office (GAO). Most of the ad hoc requests for which SAMHSA anticipates using these data require specific information on ethnicity and populations served, sometimes combining these numbers with budgetary information to estimate the costs associated with serving individuals from various demographic backgrounds or sexual orientations. SAMHSA also uses HIV program data for its own program policy, planning and development purposes.
HHS has requested that Federal agencies coordinate their efforts regarding HIV data collection and use of data to reduce burden to grantees and to better utilize collected data. To meet these requests, SAMHSA is collaborating with other Federal agencies that also have ongoing HIV programs, predominantly Health Resources and Services Administration (HRSA), Centers for Disease Control and Prevention (CDC) and National Institutes of Health (NIH), to streamline data collection efforts. For example, SAMHSA staff participated with colleagues in CDC to harmonize data collection efforts for HIV testing. SAMHSA also participates in numerous other inter-agency working groups regarding use of and reporting of HIV data. SAMHSA has already informally shared HIV testing data with CDC and HRSA and is working to harmonize HIV testing data collection so that these data may be formally shared and utilized by other Federal agencies.
CSAP has a well-established history of incorporating evaluation findings and conclusions into the policy process. Data collected through the Quarterly Progress Report and ISO instruments will provide detailed information on the effectiveness of diversely funded programs in preventing and/or reducing substance use and related problems. The information will be used to influence public policy studies and programming as they relate to the provision of youth and adult services. More specifically, the data will support the following uses by CSAP:
Annual reports to Congress
Information regarding SPF implementation and community-level change will be used in conjunction with participant-level outcome data to assess the effectiveness of currently funded prevention programs. These data will also help program planners and policy makers identify the types of strategies and combinations of strategies that are most effective in the prevention, delay or reduction of substance use, and in reducing risk factors & enhancing protective factors associated with SA and HIV transmission (e.g. knowledge, attitudes, norms, and risky sexual behaviors)
Findings concerning program inputs (intervention strategies, frequency, and length) will be used to provide program guidelines and to plan appropriate technical assistance services for programs
Findings will support CSAP publications and materials on prevention practices that are an important resource for public and private organizations involved in the design and implementation of prevention programming for youth and adults.
In sum, the findings from these reports will be a crucial resource for CSAP in setting prevention policy priorities, measuring program performance, and designing and promoting optimally effective prevention program initiatives. SAMHSA will ensure that the data on HIV positivity rates will be shared with CDC, HRSA and other relevant HHS Agencies.
CSAP plans to use the data collected through these instruments to improve program performance, meet statutory/regulatory requirements, enhance the current knowledge base on prevention programs, and increase public awareness about factors associated with substance use and HIV risk behaviors among minority populations. In order to improve program performance, CSAP shares the most recent program data with grantees, stakeholders, and other Federal officials at annual MAI grantee webinars and meetings. For example, recent performance data indicated a need to place special emphasis on abstinence from alcohol use and decrease illicit drug use rates among program participants. As a result, one of the two most recent MAI funded programs, the Capacity Building Initiative was specifically focused on providing support for infrastructure development, environmental prevention strategies, and evidence-based interventions using SAMHSA’s Strategic Prevention Framework. This change in focus will ultimately increase program performance in these areas.
CSAP plans to continue to enhance the current knowledge base on the effectiveness of prevention programs for minority populations at risk for SA, hepatitis transmission and HIV/AIDS as well as increase public awareness about factors associated with substance use and HIV risk behaviors among minority populations. Information collected under CSAP’s programs will be used by CSAP and other Federal agencies in their efforts to assess specific intervention services in the prevention or reduction of substance use and HIV/AIDS among minority populations.
Beyond HHS, CSAP plans to share lessons learned with:
The Department of Justice (DOJ) and their Office of Juvenile Justice and Delinquency Prevention (OJJDP), which funds projects that target high-risk youth and often involve SA prevention interventions.
The Department of Housing and Urban Development (HUD), which supports low-income persons and families living with HIV/AIDS through its “Housing Opportunities for Persons with AIDS Program.”
The Department of Education (DOE), one of the collaborators in the multi-agency “Safe Schools/Healthy Students" effort (focused on violence and substance abuse prevention) under the Drug Free Schools and Communities Act.
State and local program planners and the public through publications and a public-use data set.
State and local agencies also have significant responsibilities for the design and implementation of prevention programs for youth and adults. The results of the MAI findings may be useful in a variety of ways to State and local agencies, including:
Policymakers in State and local governments will have evidence of the impact of various evidence-based programs and infrastructure development models, and use of environmental strategies and information dissemination activities for changing knowledge, attitudes, norms, and systems of care associated with substance abuse and HIV/AIDS transmission in communities of color. The evidence will be useful in setting prevention policy priorities.
Program planners in State and local governments and in community-based organizations will have comparative evidence on the effectiveness of different models for the provision of youth and adult services. This information will be useful in developing funding guidelines and direct service programs.
National, nonprofit, voluntary, and professional organizations will have an accurate portrayal of the program inputs that are required to establish successful prevention approaches targeting minority and re-entry youth and adults. This information will promote optimally effective prevention program design.
Implementing evidence-based programs in minority community settings presents challenges (i.e., maintaining rigor in design and instrumentation, as well as maintaining the ability to measure impact), given the need for local adaptations for specific target populations. Using lessons learned from implemented programs, the effectiveness of specific future interventions could be enhanced.
To maximize data accuracy and reliability, the Quarterly Progress Report and ISO instruments will be web-based tools that grantees complete online. In case there is a lag between the time approval is obtained and the time resources become available for systems development, the instruments will be sent to grantees as paper questionnaires to be completed. The online tools will be made available to grantees as soon as system development is complete.
The system being planned for the instruments will require a web browser and access to the Internet. Users will be able to access the system 24 hours a day, 7 days a week, aside from scheduled maintenance windows, through the use of an encrypted username and password. Upon logging into a system-assigned account, grantees will be able to: enter data on their program; upload documents for the project officer review; and generate quarterly and annual reports of their activities. Although a large portion of the Quarterly Progress Report instrument contains data fields relevant to all MAI grantees, regardless of the program under which they are funded, there are a few fields that are relevant only to the MSI CBO grantees and some that are specific to CBI grantees. The online instruments will be designed to display only the items relevant to the funding mechanism of the respondents, based on their login information. To the extent possible, fields will be pre-filled if the response can be deduced from a response to an earlier question. Skip patterns will facilitate navigation through the instrument by only displaying items that apply to the respondent, based on information already entered into the system. The system will also allow SAMHSA’s project officers to review and approve submitted progress reports or ask the grantee to provide additional information regarding their activities. Project officers will also have the capability to generate online summary reports on their grantees’ progress.
The system design will also include a link to the participant-level instruments (baseline, exit, and follow-up surveys and dosage forms) in order to allow PEP-C’s MAI Cross-Site Team to evaluate participant outcomes within the context of the interventions that they received.
CSAP conducted an extensive literature search, consulted with staff in Federal agencies and organizations that work with substance use and HIV/AIDS prevention programs, and discussed the proposed program with substance abuse prevention experts. Specifically, CSAP:
Conducted a comprehensive literature search of completed and ongoing studies of SA and HIV/AIDS prevention programs targeting youth and adults and found insignificant duplication with this cross-site study. All studies were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or policy application of results.
Consulted with staff in CSAT, CDC, NIAID, NIDA, ACF, OJJDP, HUD, DOE and DOJ. None of these Federal organizations had collected data on prevention and early intervention programs targeting minority youth and minority re-entry youth similar to that being proposed in this submission.
Staff attended national meetings at which completed, ongoing, or contemplated evaluations were discussed and found insignificant duplication with the proposed data collection instruments.
In summary, SAMHSA did not identify any redundancy in that there were no precedents for a data collection effort similar to the one being proposed. Thus, it is clear that the data to be collected will be unique to the SAMHSA/CSAP MAI programs, collected only from MAI grantees, and not available elsewhere. In other words, the data collected through these instruments will be non-duplicative, minimize burden on grantees and SAMHSA staff alike, and will be of use to both SAMHSA, other Federal agencies within and outside of HHS, and to communities of color at large.
This data collection will have no significant impact on small entities.
The Quarterly Progress Report is a modular instrument structured around the SPF steps, and designed to be updated quarterly. Only the modules corresponding to the steps that the grantee actively worked on during any given quarter will be completed at each wave of data collection. The module on cultural competence, one of the overarching guiding principles of the SPF that affects every step, will be completed twice a year, as part of the second and fourth quarters’ progress reports. Each module contains data elements on the grantee’s accomplishments and barriers associated with the associated phase or principle of the SPF. If these data are collected less frequently SAMHSA/CSAP’s ability to promptly respond to inappropriate strategies and activities with corrective action and to meet grantees’ training and technical assistance needs in a timely fashion will be negatively affected. Delays in these responses will, in turn, have an impact on grantees’ subsequent SPF steps, causing a cascading effect on overall program effectiveness.
Another reason for quarterly reports of implementation activities is that some of the data are used to meet national data collection needs, especially on HIV/AIDS. For example, information on the numbers of individuals tested for HIV and those with positive test results are typically updated frequently to maintain as close to real-time data as possible.
The ISO data will be submitted once a year, to allow SAMHSA/CSAP to assess the success of funded programs in changing community norms and practices. Less frequent collection of these community-level data will limit the ability of project officers and the cross-site evaluation team to detect indirect service strategies that are not producing the desired effects on targeted communities and to provide timely feedback and training to grantees in modifying and enhancing their efforts.
This information collection fully complies with 5 CFR 1320.5(d) (2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on November 9, 2015 (80 FR 69234).
CSAP consulted with experts on SA, viral hepatitis (VH), and HIV/AIDS within HHS, as well as other Federal agencies with related programs or mandates, including NIDA, ACF, CDC, DOJ, OJJDP, HUD, and the DOE. Consultations resulted in the refinement of the instruments based on current Federal data reporting needs.
No payment is received by respondents.
SAMHSA has statutory authority to collect data under the Government Performance and Results Act (Public Law 1103(a), Title 31) and is subject to the Privacy Act for the protection of these data. Only aggregate data will be collected with the Quarterly Progress Report and ISO instruments, hence protecting the privacy and confidentiality of program clients and participants.
The information from grantees and all other potential respondents will be kept private and secure through all points in the data collection and reporting process. All data will be closely safeguarded, and no institutional or individual identifiers will be used in reports. It is critical to note that only aggregate data on HIV positivity will be reported to SAMHSA (e.g., 100 tests were done and 3 were positive giving 3% HIV positivity). SAMHSA and its contractors will not receive identifiable client records. A PIA and SORN application have been submitted for this data collection for review at the U.S. Department of Health and Human Services. Provider-level information will be aggregated to, at the least, the level of the grant/cooperative agreement-funding announcement.
In the Quarterly Progress Report, any data of a sensitive nature about individual clients will be aggregated to the program level (e.g., total numbers served, reached, tested, tested for the first time, and tested positive) before being reported.
The ISO instrument collects data that are not of a sensitive nature. That is, data are obtained by the grantee from available local sources such as surveys, surveillance, or administrative data, and are reported in the aggregate (e.g., as percentages or rates).
As mentioned earlier, the Quarterly Progress Report is a modular instrument that will be updated quarterly as needed. Grantees will only update the modules corresponding to the SPF steps that they actively worked on during any given quarter. The cultural competence section will be completed every other quarter. The ISO will be completed once a year. Retrospective Reporting of FY 2015 data on HIV testing activities will be required only once and only from the active grantees that were in the implementation stage of the SPF during FY 2015. These are the 50 grantees awarded in 2013 and 2014. This one-time response, annualized over three years, adds 1/3 responses per year to the burden. The retrospective reporting instrument is expected to take at most 15 minutes (0.25 hours) to complete.
Based on project officers’ previous experience with reviewing and providing feedback on grantees’ narrative progress reports, SAMHSA estimates that the Quarterly Progress Report will take, on average, four hours to complete each quarter. The ISO instrument was informally piloted among five members of the PEP-C Cross-Site Team, yielding an estimated burden estimate of two hours per response. A total of 158 grantees are expected to submit data using these instruments. All instruments will be completed by grantee staff who was typically community and social service workers. It is, therefore, appropriate to estimate the monetary value of their hour burden as the mean hourly wage for this occupational group. According to the Bureau of Labor Statistics (2015), this figure was $21.79 in May 2014.
Exhibit 1 below displays the calculation of annualized burden for the three instruments. In calculating the total number of respondents, it was assumed that a single staff member within each grant site will be the respondent for the entire data collection effort, rather than different respondents for each instrument.
Exhibit 1: Total Estimated Annualized Burden by Instrument
Type of respondent activity |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Burden Hours |
Wage Rate |
Total Hour Cost |
Quarterly Progress Report |
158 |
4 |
632 |
4 |
2,528 |
$21.79 |
$55,085 |
Indirect Services Outcomes |
158 |
1 |
158 |
2 |
316 |
$21.79 |
$6,886 |
HIV Testing Retrospective Reporting Tool |
50 |
.32 |
16 |
0.25 |
4 |
$21.79 |
$87 |
Total |
158 |
|
806 |
|
2,848 |
|
$62,058 |
The three instruments together are estimated to pose a total burden of 2,848 hours per year and to cost the grantee staff who provide the information a total of $62,062 in foregone wages.
There will be no capital, start up, or operation and maintenance costs.
The total estimated cost to the government for the data collection from FY 2016 through FY 2018 is $933,606. This includes approximately $450,000 for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and sub-recipients; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $55,602 per year represents SAMHSA costs to manage/administer the data collection and analysis for 25% each of two Social Science Analyst employees (GS-14-10, $111,203 annual salary). Approximately $105,600 per year represents SAMHSA costs to monitor and approve grantee reporting in these instruments (10% time of 10 Project Officers at $105,600 annual salary). The annualized cost is approximately $311,202
This is a de novo submission for instruments developed from templates previously in use for progress reporting and program monitoring purposes. Currently there are 0 burden hours in the OMB inventory.
Analysis Plans
As previously noted, the main purpose of the quarterly progress reports is to provide SAMHSA/CSAP with timely information about the progress of the MAI grantees through the SPF steps, to identify and promptly respond to training and technical assistance needs of the grantees, and to recommend corrective action in a timely fashion in cases where grantees’ activities do not comply with the SPF. Some of the data fields in the Quarterly Progress Report are also used to meet SAMHSA’s reporting requirements. A third function of the Quarterly Progress Report data is to provide background information for the MAI National Cross-Site Evaluation. The data collected through the ISO will be assessed to monitor community-level change in norms and practices in areas exposed to MAI-funded environmental strategies and information dissemination activities. In the rest of this section, planned analysis methods for each of these functions is discussed separately.
The utilization of the data for grant management and monitoring purposes involves narrative and qualitative reviews of the information rather than quantitative analyses. POs qualitatively compare the quarterly performance of each grantee to their performance in previous quarters and to the expected progress trajectory suggested by the SPF model. Grantees that are not performing as expected are provided with customized technical assistance from a variety of sources depending on the nature of the specific challenges.
There are three data fields in the Quarterly Progress Report that are used to fulfill SAMHSA’s reporting requirements: (i) total number of individuals tested for HIV; (ii) number of individuals who tested positive; and (iii) number of tested individuals who were homeless at the time of the service. The first and third of these numbers will be reported directly from the submitted report with no further calculations. The second will require the calculation of a percentage (positivity rate). That is the extent of the analysis needed to fulfill the reporting requirements.
The cross-site evaluation will utilize these data in several ways. First, information about each implemented direct-service intervention will be linked to the pre-post and follow-up survey data collected from the participants of the information, in order to address questions about the relative effectiveness of different prevention strategies and combinations of strategies at the individual level. Second, the community-level data collected annually from each grant site using the ISO will be combined with information about the implemented indirect-service interventions to assess the effectiveness of these interventions in producing community-level changes in norms, systems, and practices.
The proposed analysis utilizing the Quarterly Progress Report and ISO data includes several distinct steps:
Descriptive analysis of grantee targets, organizational structure, training and technical assistance activities, and implemented interventions will be conducted and the results presented separately by the Funding Opportunity Announcement (FOA) to which the grantees responded.
Pooled analyses of participant-level outcomes will be conducted to assess overall program effects and their sustainability. Program effects will be evaluated through paired comparisons of baseline and exit values. Sustainability of effects will be evaluated through paired comparisons of baseline and follow-up values. Past analyses have suggested that some measures continue to improve after program exit. To continue to assess this post-exit improvement, paired comparisons between exit and follow-up values will also be conducted.
Hierarchical linear modeling will be used to measure program effects on participants while accounting for the interclass correlations among participants of the same grantee. These models will be expanded to identify and control for moderating factors at the participant level. This multivariate analysis will yield information about variations in outcomes due to the sociodemographic characteristics and baseline risk levels of individual participants.
Site-specific data obtained from the Quarterly Progress Report, such as types and combinations of interventions implemented, fidelity of implementation, and grantee organizational characteristics, will be introduced into the multilevel models to investigate the sensitivity of effectiveness models to differences in intervention characteristics, fidelity, and grantees’ organizational characteristics. The planned multilevel multivariate models will also test hypotheses about interactions between individual and site-specific factors in determining participant outcomes.
At the community level, data collected through the ISO will be analyzed to assess improvements in community outcome measures. Each year’s value for each submitted outcome measure will be compared to its baseline value to identify statistically significant changes. Regression-based linear models will also be estimated, with community-level effect size (such as odds-ratios) as the dependent variable and intervention characteristics (as reported in the Quarterly Progress Report) as independent variables. This analysis will provide information about intervention features that are significant predictors of positive community-level outcomes.
Analysis Techniques and Statistical Test Determination
In assessing overall participant- and community-level improvement based on pooled data, paired comparison tests appropriate to the level of measurement will be employed. For dichotomous outcome measures, significance will be tested using McNemar’s test (Lidell, 1976; Yang, Sun, & Hardin, 2010). For normally-distributed continuous outcome measures, matched-pairs t-tests will be used to assess significance. The significance of change in ordinal or skewed continuous measures will be tested using the Wilcoxon signed-rank test (Wilcoxon, 1945; Blair & Higgins, 1980).
A multi-level analysis approach (e.g., HLM) will be used to investigate the effects of participant- and grantee-level characteristics on participant outcomes. Site-specific characteristics hypothesized to have a bearing on program effects, such as choice of prevention strategies and type of grantee organization, will be included in the dataset together with participant-level baseline, exit, and follow-up survey records and dosage data. Nesting participant-level data within program-level data in this fashion will allow the construction of multi-level causal models that simultaneously test for the effects of participant and program characteristics on program outcomes and to identify significant interactions between these two levels, while accounting for the hierarchical data structure (Osborne, 2000; Raudenbush & Bryk, 2002).
Reporting and Dissemination Plan
In addition to the annual cross-site reports and presentations of results to SAMHSA project officers and grantees, the MAI cross-site evaluation results will be made available to the public through publications and conference presentations. The following journals carry articles on SA prevention and HIV/AIDS and are expected to serve as potential vehicles for distribution of evaluation results: Journal of Substance Abuse Treatment, International Journal of Addictions, Journal of Community Psychology, Journal of Adolescent Research, Journal of Adolescent Health, Preventive Medicine, Evaluation Review, Policy Studies Review, and The American Journal of Public Health. Evaluation results could also be published in other journals that focus on HIV/AIDS. These include The Journal of the American Sexually Transmitted Disease Association, Health Education and Behavior, AIDS: Official Journal of the International AIDS Association, AIDS Education and Prevention, The Journal of Sex Research, AIDS Care, Psychological and Socio-Medical Aspects of AIDS/HIV, and Current Opinion in HIV and AIDS. Evaluation results are also targeted for publication in journals focusing on infectious diseases. These include, among others, The Journal of the American Microbiological Association and Journal of Infectious Diseases.
The evaluation results will be distributed through presentations at annual conferences of national and international public health, prevention, and program evaluation organizations, such as the Society for Prevention Research, the American Public Health Association, the National Association of Alcohol and Drug Abuse Counselors, The National Prevention Network, the American Evaluation Association, The Society for Prevention Research, and HIV/AIDS national meetings as well as regional and State SA prevention and treatment associations. HIV/AIDS meetings could include, among others, CDC Annual Conferences on AIDS and Conferences of the International AIDS Society. Results could also be presented at meetings focusing on infectious diseases such as annual meetings of the American Society of Microbiology.
Documents will also be prepared and published on behalf of the government (CSAP) through the Government Printing Office (GPO) for Federal agency and public use. Findings will also be available via OMB’s Website: www.expectmore.gov, as well as in annual reports to Congress and the performance detail sections of annual SAMHSA budgets as they become publicly available.
Timeline
MAI’s MSI CBO grants are three-year programs while the CBI grants are funded up to five years. Typically, there is an initial period devoted to Steps 1, 2 and 3 of the SPF, namely conducting needs assessment, capacity building, and strategic planning. Grantees begin implementation and data collection only after their proposed strategic prevention plans are approved by their SAMHSA project officers. The approximate dates of the SPF milestones for the grantees that will be using the instruments are presented in Exhibit 2 below.
Exhibit 2. Project Timelines
|
MSI CBO Awarded in 2013 (3-year grants) |
MSI CBO Awarded in 2014 (3-year grants) |
MSI CBO Awarded in 2015 (3-year grants) |
CBI Awarded in 2015 (5-year grants) |
Needs Assessment, Capacity Building, Strategic Planning |
10/2013 – 6/2014 |
10/2014 – 6/2015 |
10/2015 – 6/2016 |
10/2015 – 6/2016 |
Implementation, Data Collection |
6/2014 – 9/2016 |
6/2015 – 9/2017 |
6/2016 – 9/2018 |
6/2016 – 9/2020 |
Follow-up, Evaluation, and Reporting |
Ends 12/31/2016 |
Ends 12/31/2017 |
Ends 12/31/2018 |
Ends 12/31/2020 |
The expiration date will be displayed.
No exceptions are required. All certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | HIV OMB Supporting Statement |
| Author | proddy |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |