OMB Supporting Statement:
Minority Substance Abuse/HIV Prevention Initiative
A. Justification
1. Circumstances of Information Collection
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP), is requesting approval from the Office of Management and Budget (OMB) to obtain clearance for the Youth and Adult Minority Substance Abuse (SA)/HIV Prevention Initiative Questionnaires to conduct a cross-site evaluation of the Minority SA/HIV Prevention Initiative (MAI) funded by SAMHSA’s CSAP from September 2005 to September 2010. Contingent upon the availability of funding, CSAP plans to fund additional cohorts in FY 2008, 2009 and 2010.
Since the AIDS epidemic began, injection drug use (IDU) has accounted for more than one-third of AIDS cases in the United States. Of the 43,517 new cases of AIDS reported in 2000, 25 percent were injection drug use (IDU)-associated (Summary Fact Sheet on HIV/AIDS, 2008). Racial/ethnic minorities in the U.S. are most heavily affected by IDU-associated AIDS. In 2000, African American adults and adolescents accounted for 26 percent of IDU-associated AIDS cases and Hispanic adults and adolescents accounted for 31 percent, as compared to 19 percent of all IDU-associated AIDS cases among their white counterparts. IDU-associated AIDS accounts for a larger proportion of cases among women than among men. Fifty-seven (57) percent of all AIDS cases reported among women have been attributed to IDU or sex with partners who inject drugs as compared with 31 percent of cases among men.
Despite the decline in AIDS cases in certain populations and regions resulting from improved HIV treatment, 2003 data reported by CDC indicate that more people are living with HIV/AIDS than ever before (CDC, 2008). CDC estimates that about 1 million people in the United States are living with HIV or AIDS and about one quarter of these people are unaware of their infection, which puts them and others at risk. Groups at highest risk since the beginning of the epidemic include men having sex with men (MSM) and IDUs. Other groups who are also at high risk for HIV transmission include people of color, women (particularly Black and Latina/Hispanic women), and youth (Office of Applied Studies, 2006).
Although several Federal agencies have mandates to fund projects targeting minority and minority re-entry populations who are at risk for substance use, HIV/AIDS, and hepatitis, very little is known about the efficacy of such programs once they become widely disseminated. Prior efforts to evaluate government substance use prevention initiatives targeting at-risk populations have focused on highly specific program models and strictly defined target groups, or have been hampered by lack of valid questionnaires and poor study design. Once these models are more widely disseminated to community-based agencies (which typically implement these programs under less rigorous and controlled parameters), measures and efforts to adequately assess outcomes have not been sufficiently designed to determine the true impact of these interventions. In addition, the link between SA and HIV/AIDS and hepatitis outcomes has not been evaluated for these programs or in local community settings.
Literature searches within SAMHSA’s three centers and with five other Federal agencies were conducted to identify studies with similar goals and expected outcomes. No similar study has been conducted that examines prevention initiatives regarding SA in relation to HIV/AIDS and hepatitis. Even though the present cross-site evaluation is unique from others that have been conducted in the field, information generated from these literature searches has sharpened the present cross-site evaluation design and enhanced the likely utility of the results.
The cross-site evaluation employs measures to safeguard the privacy and security of participants’ responses, and supports the program and study needs of multiple Federal agencies. Sample size, respondent burden, and intrusiveness have been minimized to be consistent with the cross-site evaluation objectives. Every effort has been made to coordinate cross-site data collection with local data collection efforts in order to minimize respondent burden, including pilot testing, which has assisted in controlling burden and ensuring the user-relevance of questions.
The cross-site evaluation results will have significant implications for the SA, HIV/AIDS and hepatitis prevention field, the allocation of grant funds, and other evaluation activities conducted by multiple Federal, state, and local government agencies. The results could be used to develop Federal directives in support of SAMHSA/CSAP program initiatives, inform the public of lessons learned and findings, improve existing programs, and promote replication and dissemination of effective prevention strategies.
This program is authorized by Section 516 of the Public Health Service Act, as amended, and subject to the availability of funds. It was supported by the Congressional Black Caucus through its Conference Report on H.R. 4328, Making Omnibus Consolidated and Emergency Supplemental Appropriations Act, for FY 1998 (House of Representatives, October 19, 1998), to address prevention and treatment needs of minority communities that are heavily affected by HIV/AIDS and hepatitis. It builds on previously authorized programs addressing these issues (discussed below). In addition, the program addresses the DHHS’s Healthy People 2010 goals and objectives 26-1, 26-3, 26-7, 13-7, 13-8, and 25-12, the Twenty Department –Wide Objectives regarding SA prevention, and SAMHSA’s HIV Matrix priorities.
While the program will serve around 35,000 adults and youth through evidence-based programs and strategies, questionnaire data will be collected on about 3,800 participants. These will be the participants who receive direct services for over 30 days at three time points: (1) baseline (program entry), (2) program exit, and (3) three to six months post-exit. It is estimated that about 88 % of program participants will be adults and 12% youth. The CSAP National Outcome Measures (NOMs) Adult and Youth questionnaires, which have been approved by OMB (OMB # 0930-0230) for use in all CSAP evaluation studies, will be used to measure alcohol, tobacco, and other drug (ATOD) use and risk factors associated with ATOD use among program participants. These NOMs data are used to report on Government Performance and Results Act (GPRA) and Performance Assessment Rating Tool (PART) findings across CSAP programs. For this program, these cross-site questionnaires are augmented with additional scales to measure other important risk and protective factors uniquely associated with HIV/AIDS and hepatitis among minority populations and minority re-entry populations in communities of color. Each questionnaire contains 135 questions, of which 102 relate to HIV/AIDS and hepatitis. Grantees will report these data online through the CSAP Services Accountability Monitoring System (CSAMS).
Background
CSAP serves as SAMHSA’s focal point for SA prevention initiatives among youth and other high-risk populations as mandated by Section 517 of the Public Health Service (PHS) Act (42 USC 290bb-23).
Epidemiological studies on the dynamics of SA, HIV/AIDS, and hepatitis demonstrate a continued need to reach out to communities of color, particularly to those reporting high rates of HIV/AIDS, hepatitis, and sexually transmitted diseases (STDs). The Centers for Disease Control and Prevention (CDC)’s Morbidity and Mortality Weekly Report indicates that approximately 37,000 persons were diagnosed with HIV/AIDS in 2005, of which 49% were Black (not Hispanic), 18% Hispanic, 1% Asian-Pacifier Islanders, 1% American Indian and Alaskan Native, and 3% White (CDC. MMWR, November 18, 2005, 1153). CDC also reported that more than one-third of all HIV infections progressed to AIDS within 12 months after diagnosis. In 2005, CDC reported IDU as the exposure category for 14 % of male and 19% of female new AIDS cases. Due to similar transmission routes, many minority populations at risk for HIV/AIDS are also at risk for viral hepatitis. For both HIV/AIDS and viral hepatitis, IDU individuals are an important target population. Between 50% and 90% of persons infected with the hepatitis C virus (HCV) report IDU as the risk (CDC, 2005).
No matter how they are transmitted, HIV/AIDS and viral hepatitis are infectious diseases that have drastic long-term medical, economic, and social consequences for minority populations. Meeting the challenges posed by viral hepatitis requires close coordination with existing local, state, and territorial SA and HIV/AIDS programs. To this end, SAMHSA is working to improve access to quality SA, HIV/AIDS, and hepatitis prevention services by increasing outreach and service capacity to at risk populations of color. Use of the Strategic Prevention Framework (SPF) as a method to prevent and reduce both SA and the transmission of HIV/AIDS and hepatitis will lay the necessary foundation for effective and sustainable prevention service delivery in the context of SA and HIV/AIDS.
Of particular concern to communities of color is the return of ex-offenders, otherwise known as the re-entry population. Despite the efforts of correctional facilities to prevent sexual risk-taking behavior and SA among incarcerated persons, a significant number engage in high-risk activities (such as IDU, tattooing, and coerced sexual activity), placing others at risk for HIV and hepatitis transmission. Each year, many of these persons, unaware of their HIV and hepatitis status, return to their communities and re-engage in SA and other high-risk behaviors, putting others at an even greater risk for HIV/AIDS and hepatitis transmission.
In FY 2005, the current MAI Cohort 6 Program funded 81 five-year grants to community-based organizations. At this time the number has dropped to 80 since one grant was closed out. This program combines planning and services funding and requires all grantees to participate in this cross-site study. They are expected to provide leadership and coordination on the planning and implementation of the SPF that targets minority populations and the minority re-entry population in communities of color with a high prevalence of SA, HIV/AIDS, and hepatitis. The primary objectives of the cross-site study are to:
Assess the process of adopting and implementing the SPF with the target population,
Measure the effectiveness of specified intervention strategies such as cultural enrichment activities, educational and evaluation, vocational resources, or computer-based curricula, and
Determine the success of the MAI in delaying, preventing, and/or reducing the use of ATOD among the target population.
This MAI Cohort 6 program builds on five previous SAMHSA/CSAP SA/HIV Prevention grant programs that provided SA and HIV/AIDS planning and prevention services for minority populations. These were:
HIV Cohort 1 Services Grants – 48 funded in FY 1999 for 3 years
HIV Cohort 2 grants funded in 2001:
Planning Grants – 41 funded for 1 year
Expansion Grants – 20 funded for 3 years
Youth Services Cooperative Agreements – 18 funded for 3 years
HIV Cohort 3 Targeted Capacity Expansion Initiative for SA and HIV Prevention in Minority Communities Grants funded in 2001:
Planning Grants – 10 funded for 1 year
Services Grants – 48 funded for 3 years
HIV Cohort 4- Targeted Capacity Expansion Initiative for SA and HIV Prevention in Minority Communities Grants funded in FY 2003:
Planning Grants – 10 funded for 1 year
Services Grants – 22 funded for 5 years
HIV Cohort 5 Services Targeted Capacity Expansion Initiative for SA and HIV Prevention in Minority Communities Grants - 46 grants funded in FY 2004 for 4 years
As noted, the goal for Cohort 1 grants was to provide services. Goals for Cohorts 2-5 grants were to add, increase, or enhance integrated SA and HIV prevention services by providing supportive services and by strengthening linkages between service providers for at-risk minority populations. HIV Cohort 6 will add to the prevention knowledge base by examining the effectiveness of prevention services for minority populations and the minority re-entry population in communities of color with a high prevalence of SA, HIV/AIDS, and hepatitis.
These past programs have enabled CSAP to make great progress in providing innovative, community-based drug prevention, planning, and intervention services to minority populations at risk for SA and HIV/AIDS. The results from Cohort 3 have demonstrated reductions in ATOD use, reductions in risky sexual behavior, and an increase in education pertaining to HIV/AIDS and hepatitis prevention. Between baseline (n=80) and exit (n=131), there was a significant decrease in the number of participating youth who reported having sex after getting drunk or high, and a significantly higher number of participants taking HIV/AIDS education classes at exit (n=2,737) than at baseline (n=1,237). Among Cohort 3 adult participants, the number smoking cigarettes decreased from baseline (n=194) to exit (n=176), and the number drinking alcohol decreased from baseline (n=85) to exit (n=59).
The Cohort 6 MAI Program differs substantially from these earlier programs in that it targets a very different population and calls for the use of the SPF and evidence-based programs. While these grantees have substantial flexibility in choosing evidence-based programs, they are all required to base their projects on the five steps of SAMHSA’s SPF to build service capacity specific to SA, HIV/AIDS, and hepatitis prevention (HP) services. In FY 2006, all the grantees initiated Steps 1, 2, and 3 of the SPF, namely conducting a Needs Assessment, Building Capacity, and Implementation Planning. Once their plans have been approved by their Project Officers they can proceed to Step 4 - Implementation, and then to Step 5 - Conducting the Evaluation.
2. Purpose and Use of Information
The purpose of this data collection and analysis is to conduct a cross site evaluation of the impact of the program. The knowledge about program effectiveness will help with the prevention programs for minority populations at risk for SA, HIV/AIDS, and hepatitis, and provide evidence and conclusions for disseminating optimally effective prevention policies and programs.
Information collected will be used by CSAP, SAMHSA, and other Federal agencies in their efforts to assess specific intervention services in the prevention or reduction of substance use, HIV/AIDS, and hepatitis among minority community and re-entry populations in communities of color across the Nation. Policymakers need to learn how to reach more minority populations and minority re-entry populations residing in communities of color that are heavily affected by HIV/AIDS and hepatitis. CSAP wishes to continue to enhance the Nation’s impact on the HIV/AIDS epidemic with the current cross-site evaluation.
CSAP is seeking approval from OMB to expand the NOMs youth and adult questionnaires (OMB No. 0930-0230) by adding items unique to the MAI program and necessary to conduct a systematic cross-site analysis of the intervention and/or infrastructure development services provided by grantees supported by this program. This cross-site evaluation will involve not only collecting information on the planning and delivery of the evidence based programs, but also assessing their effectiveness. Grantees are conducting on-going monitoring and analysis of their projects to assess program effectiveness, including Federal reporting of the 1993 GPRA, PART, and SAMHSA/CSAP National Outcome Measures (NOMs), as well as HIV Counseling and Testing.
CSAP will also share the outcome information and lessons learned with other Federal DHHS agencies, including but not limited to SAMHSA’s Center for Substance Abuse Treatment (CSAT), the National Institute on Drug Abuse (NIDA), the Center for Communicable Diseases (CCD), and the Administration for Children and Families (ACF), which administers several drug-related programs targeted at hard-to-reach and at-risk populations. Beyond DHHS, CSAP plans to share the outcomes and lessons learned with:
The Department of Justice (DOJ) and their Office of Juvenile Justice and Delinquency Prevention (OJJDP), which funds projects that target high-risk youth and often involve SA prevention interventions,
The Department of Housing and Urban Development, which supports two major initiatives (the National Youth Sports Program and the Public Housing Drug Elimination Program) that target youth at risk of substance use and provide positive alternative activities for at-risk youth in a drug-free environment.
The Department of Education, which funds the Safe Schools/Healthy Students effort (focused on violence and substance abuse prevention) under the Drug Free Schools and Communities Act.
State and local program planners and the public through publications and a public-use data set.
Implementing evidence-based programs in minority community settings presents challenges, including maintaining rigor in design and instrumentation, as well as maintaining the ability to measure impact, given the need for local adaptations for specific target populations. Using the lessons learned from the previous programs, the current focus is on enhancing the effectiveness of specific interventions for reducing risk factors and/or enhancing the factors that protect against ATOD abuse and HIV/AIDS among both minority youth and adults and minority youth and adult re-entry populations.
CSAP has a history of strong efforts to incorporate evaluation findings and conclusions into the policy process in a variety of ways. This study is designed to provide more specific information on the effectiveness of diversely funded programs in preventing and/or reducing ATOD use and related problems. CSAP will use the evaluation results to influence public policy, research, and programming as they relate to the provision of youth and adult services. More specifically, the research will support the following uses by CSAP:
Findings will be used in required GPRA and PART performance reporting, and will be presented in annual Reports to Congress.
Findings regarding SPF implementation will be used to delay or reduce ATOD use, influence positive sexual behaviors, change ATOD attitudes, and reduce associated problem sexual and substance use behaviors in participants, and as well as to assess the effectiveness of currently funded prevention programs. Furthermore, the common use of ATOD outcome measures (from CSAP, GPRA, PART, and NOMs) will allow CSAP to compare initiatives (including the previous HIV/AIDS programs). Such extensive cross-initiative information will be used to set broad prevention policy priorities.
Findings concerning the ATOD and sexual behavior risk factors as both program outcomes and mediating factors will be used by CSAP to refine policy and shape future program funding announcements. In addition, CSAP may use the findings to provide recommendations to States regarding selection of evidence-based programs since a portion of Block Grant monies given to the States must be spent on substance abuse prevention. Additional monies have been awarded to some states through the State Incentive Grants (SPF-SIG)
Findings concerning program inputs (intervention strategies, frequency, and length) will be used by CSAP to provide program guidelines (e.g., through RFA’s) and to plan appropriate technical assistance services for programs/States.
Findings will support CSAP publications and materials on prevention practices that are an important resource for public and private organizations involved in the design and implementation of prevention programming for youth and adults.
In sum, the findings from the study will be a crucial resource for CSAP in setting prevention policy priorities, measuring performance, and in designing and promoting optimally effective prevention program initiatives. Although the study is designed primarily to address CSAP program requirements, the cross-site study results will be useful to other Federal, State and community agencies involved in efforts to prevent or reduce ATOD use among youth and adults. While some of these agencies are specifically interested in providing preventive health services, others have a more general interest in approaches or strategies that have been proven effective. The following are illustrative of the potential uses of data from the MAI cross-site study by Federal organizations and agencies outside of CSAP:
The Center for Substance Abuse Treatment (CSAT) may use the data and findings to design youth and adult treatment programs that address those risk factors most strongly related to ATOD abuse and HIV/AIDS.
The National Institute of Allergy and Infectious Disease (NIAID) may use findings to inform and refine their own basic research programs in prevention of HIV/AIDS and hepatitis. The MAI cross-site study will identify particularly promising intervention strategies addressing risk, resiliency, ATOD use, and HIV/AIDS. Study findings will be useful for targeting future controlled studies of prevention interventions.
The National Institute on Drug Abuse (NIDA) may use findings to inform and refine its own basic research programs in prevention. The MAI cross-site study will identify particularly promising intervention strategies addressing risk, resiliency, ATOD use, and HIV/AIDS. Study findings will be useful for targeting future controlled studies of prevention interventions.
The Administration for Children and Families (ACF) may use study findings to guide its research and programmatic efforts involving children and youth. The findings will also help the Agency in its planning of future services funded under The Drug Abuse Prevention Program for Runaway and Homeless Youth and The Youth Gang Prevention Program.
The Department of Education (DOE) disseminates findings on effective intervention strategies to schools to help in the development of programs. Findings from this MAI may be used to provide local educational agencies with information necessary to enhance the success and effectiveness of after-school programs for youth.
The Department of Housing and Urban Development (HUD) may use the MAI cross-site findings to improve the program designs of both the Public Housing Drug Elimination and Youth Sports Programs. More importantly, HUD has an interagency agreement with DOJ to conduct ATOD prevention training for public housing authorities, which uses the risk and resiliency factor model.
The Department of Justice may use the study findings to guide its program efforts to prevent and treat HIV/AIDS transmission among prison inmates and in serving its juvenile justice population.
State and local agencies also have significant responsibilities for design and implementation of prevention programs for youth and adults. The results of the MAI cross-site findings may be useful in a variety of ways to State and local agencies:
Policymakers in State and local governments will have evidence of the impact of various evidence-based programs and infrastructure development models on preventing or reducing ATOD use and HIV/AIDS and hepatitis among minority and re-entry youth and adults residing in communities of color. The evidence will be useful in setting prevention policy priorities.
Program planners in State and local governments and in community-based organizations will have comparative evidence on the effectiveness of different models for the provision of youth and adult services. This information will be useful in developing funding guidelines and direct service programs.
National, not-for-profit and nonprofit voluntary, and professional organizations will have an accurate portrayal of the program inputs that are required in successful programs targeting minority and re-entry youth and adults residing in communities of color. This information will promote optimally effective prevention program design.
3. Use of Information Technology
It is anticipated that technical infrastructure and data management skills will vary across grantee sites. To maximize data accuracy and reliability, online data entry tools are designed for the questionnaires being submitted for clearance. These tools will be made available to grantees through CSAP’s Services Accountability and Monitoring System (CSAMS) Web portal. The tools are designed to reflect the structure of the questionnaires, and to allow the entry of data from completed questionnaires directly into the system through the use of radio buttons corresponding to response options. The system automatically quantifies the selected response options and stores the numeric codes in a SQL server for subsequent extraction, cleaning, and analysis.
CSAMS is maintained by CSAP’s Data Information Technology Infrastructure Center (DITIC). The data entered online by grantees are periodically extracted by DITIC and transmitted in encrypted form to CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) for cleaning, record linkage, and analysis. Grantees have two options for accessing the data they enter online. In the first option, grantees can download, in spreadsheet form, the raw data they have entered online as soon as it is submitted. Grantees can also access their data from the cleaned analysis files prepared by DACCC which are posted on CSAMS under password protection.
Grantees that prefer to create their own data files have the option of uploading complete data files to CSAMS. A grantee choosing this data submission option is required to use a standard codebook while preparing the data, thus ensuring that uploaded data files have the same numeric coding and variable naming conventions as the data entered using the online tools. The online data entry tools reduce the grantees’ burden by facilitating the data entry process and minimizing coding and variable naming errors. The tools allow grantees without access to data management/analysis software to accurately quantify the information in completed questionnaires. The DACCC will then conduct cross-site analyses to determine outcomes for the program as a whole.
The CSAP multi-site questionnaires have been developed and used by grantees in previous HIV cohort programs and have been demonstrated to work efficiently and effectively. Based on the feedback of the HIV pilot, the questionnaires and procedures for electronic transmission of data files have been improved to increase efficiency and minimize burden on both training participants and grantee staff.
4. Efforts to Identify Duplication
CSAP conducted an extensive literature search, consulted with staff in Federal agencies and organizations that work with ATOD, HIV/AIDS, and hepatitis prevention programs, and discussed the proposed program with SA prevention experts. Specifically, CSAP:
Conducted a comprehensive literature search of completed and ongoing studies of ATOD and HIV/AIDS prevention programs targeting youth and adults and found insignificant duplication with this cross-site study. All studies were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or application of results. The search found that there has been no outcome evaluation of substance prevention programs of comparable scope to this study — no study spanning SA, HIV/AIDS, and hepatitis for minority youth and minority re-entry youth residing in communities of color — and no study has used the same measures as those proposed to document outcomes across youth and adults. As mentioned previously, evaluations of previous CSAP HIV Cohorts have been conducted. These past program designs suffered from severe methodological flaws which limited the ability to detect outcomes and produce meaningful results. The present initiative seeks to improve upon the lessons learned by applying more rigorous analytical procedures and a more tailored design.
Consulted with staff in CSAT, CDC, NIAID, NIDA, ACF, OJJDP, HUD, DOE and DOJ. None of these Federal organizations has conducted a cross-site outcome evaluation of prevention and early intervention programs targeting minority youth and minority re-entry youth residing in communities of color that is similar to the one being proposed for this study.
Staff attended national meetings at which completed, ongoing, or contemplated evaluations were discussed and found insignificant duplication with the proposed study.
In summary, CSAP did not identify any redundancy in that there were no precedents for a cross-site study of projects like the one being proposed. The studies that they did find were limited to general populations, HIV/AIDS alone, or HIV/AIDS and SA but not hepatitis, not all three conditions. Others suffered severe methodological problems such as low sample sizes, and/or lack of published/reliable/valid measures and scales that make it unlikely that current information will be published or released among the scientific community or in respected journals, government publications, etc. Thus, it is clear that the data to be collected are unique to the CSAP MAI programs, are collected only for the CSAP programs, and are not available elsewhere. The data collected through the multi-site effort will be non-duplicative, minimize burden on respondents, and be of use to both CSAP and the communities of color.
5. Involvement of Small Entities
Most HIV grantees are small entities. However, each grant includes funding for evaluation and data collection. Therefore, we anticipate that data collection will not impose a substantial burden on grantees.
6. Consequences if Information is Collected Less Frequently
The data will be collected from participants at three points in time. All three points in time are essential, as it will demonstrate whether sustainable results can be achieved over time after the program has ended, and if so, for which types of interventions and populations. Failure to collect the information from all participants at all three points in time will result in missed opportunities and lessons learned on how to provide a quality improvement mechanism for CSAP to continually monitor and refine its prevention programs to ensure they meet the needs of minority populations and minority re-entry populations at risk for SA/HIV/AIDS and hepatitis residing in communities of color.
Without this information:
CSAP will not be able to determine the extent to which it can prevent, reduce, and/or delay SA and in turn reduce other risky behaviors that can lead to HIV/AIDS and hepatitis infection among minority populations and minority re-entry populations residing in communities of color.
CSAP will not be able to monitor the quality of its prevention programs and determine how they can be improved to ensure continued success at meeting the needs of minority populations at risk for HIV/AIDS and hepatitis.
CSAP will not be able to describe fully the range of prevention services used and the efficacy of evidence-based programs.
CSAP will not be able to ascertain if participants are more knowledgeable about HIV/AIDS and hepatitis and how they relate to substance abuse as a result of program participation.
CSAP will not be able to identify those prevention services that are most effective and identify the potentially unique needs of minority populations residing in the community and minority re-entry populations.
CSAP will not be able to meet its Federal reporting requirements to DHHS, OMB, and Congress.
7. Consistency with the Guidelines in 5 CFR 1320.5(d) (2)
This information collection fully complies with 5 CFR 1320.5(d) (2).
8. Consultation Outside the Agency
a. Federal Register Notice
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on Wednesday, August 22, 2007, on pages 47055-47056. No comments were received.
CSAP has consulted experts both within and outside the Agency on refinement of the design, instrumentation, products, and statistical aspects of the cross-site study at critical junctures during the study design. These consultations provided an opportunity to: obtain advice and recommendations on the identification and prioritization of the information to be gathered; ensure the technical quality, appropriateness, and user relevance of the study results; verify the importance, relevance, and accessibility of the information to be sought; and minimize respondent burden.
b. Consultations Within the Agency:
The multi-site study and questionnaire design were based on initial consultation with SAMHSA experts from CMHS and CSAT, and on pilot testing with the previous HIV Cohorts.
CSAP consultation was provided by CSAP representatives noted in Table 1:
Table 1: CSAP Representatives
Nikki Bellamy, Ph.D. Previous Position: PEC Project Officer Division of Knowledge Application and Systems Improvement Current Position: Public Health Advisor Center for Mental Health Services (240) 276-2418 |
Paul Brounstein, Ph.D. Previous Position: Director, Division of Knowledge Application and Systems Improvement Current Position: The Urban Institute Washington, DC (202) 833-7200 |
Other SAMHSA HIV program experts, such as Jenifer Fiedelholtz of OPPB and David R Robertson of CSAT, among others, were consulted on the following issues:
Draft study design plan and privacy/data security procedures; plan for coordinating and collecting data; measures to be used to assess outcomes; and mediating factors
Suitability of proposed assessment questionnaires
Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site study and reporting study findings
Means of minimizing the burden on project staff and program participants
Identification of efforts to ensure user relevance of results.
c. Consultations Outside of the Agency
CSAP consulted with other experts on SA, HIV/AIDS, and hepatitis as well as with other Federal agencies with related programs or mandates. These consultations resulted in the refinement of measures and the coordination of Federal data needs. To assess the effectiveness of the programs funded since 2001, CSAP funded a Program Evaluation Center (PEC) to conduct a cross site evaluation. An Evaluation Research Group was established to work in collaboration with CSAP and the PEC to develop and pilot test the questions used for this study. The following individuals were consulted:
Table 2: Outside Consultants
Minority Substance Abuse and HIV Prevention Initiatives Evaluation Research Group (ERG) Members |
|
Outside Consultants |
|
Faye Belgrave, Ph.D. (Grantee Evaluator) Virginia Commonwealth University Department of Psychology Richmond, VA (804) 225-4415
|
Carol Odo, Ph.D. (Grantee Project Director) Ke Ola Mamo Honolulu, HI (808) 550-0885
|
Edwin Gonzalez-Santin, M.S.W. (Grantee Evaluator) Office of American Indian Projects School of Social Work Arizona State University Tempe, AZ (480) 965-5156
|
Teresa Shattuck, Ph.D. (Grantee Evaluator) Shattuck and Associates Mt. Airy, MD (301) 829-5737
|
Wade Nobles, Ph.D. (Grantee Project Director) Institute for Black Life Oakland, CA (510) 836-3245
|
Rodolfo Vega, Ph.D. (Director of PEC for Cohort I) JSI Research & Training Institute, Inc 44 Farnsworth Street Boston, MA (617) 482-9485
|
PEC Representatives |
|
Elizabeth Harris, Ph.D. (PEC Co-Project Director) EMT Associates, Inc. (818) 990-8301
|
J. Fred Springer, Ph.D. (PEC Senior Analyst) EMT Associates, Inc. (615) 595-7658
|
Jack Hermann, Ph.D. (PEC Co-Project Director) ORC Macro (301) 572-0336
|
|
The issues discussed were:
The evaluation design and data privacy procedures
The plan for coordinating and collecting data and measures to be used to assess outcomes and mediating factors, including:
Suitability of proposed assessment questionnaires
Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site evaluation and reporting study findings
Means of minimizing the burden on project staff and program participants
Efforts to ensure user relevance of results.
9. Payment to Respondents
Individual grantees may provide incentives to respondents for their participation in the cross-site evaluation. Incentives are offered since this is a hard to reach population. Each grantee determines the cost and type of incentive for its participants as appropriate to its local culture and context. Incentives include vouchers, gift cards, and coupons to purchase items valued at $20.00 for their participation. Grantee efforts to provide incentives are guided by SAMHSA Extramural Policy Statement No. SEPS 06-02: Use of Discretionary Grants Funds for Incentives to Program Participants.
10. Assurance of Confidentiality
CSAP has designed the multi-site questionnaire data collection strategy so that no identifying information such as names or complete Social Security Numbers will be requested of participants. This strategy is described in the following paragraphs:
A ten (10) digit unique identification number (ID) is used on the forms in order to track the responses of an individual over time and across grantees. Each participant’s name and unique 10-digit ID are written on the face (cover) sheet of the survey and the same 10 digit ID entered on page 1 of the instrument. This is completed by the Administrator prior to handing the instrument to the participant. Participant names must not be written on any other page but the face sheet. The face sheet is then filed in a locked storage unit separate from the survey forms. The 10-digit ID has the following components:
Grantee Site Identifier (A): Each grantee has
been assigned a site identification number by CSAP. The site
identification numbers range from 601 to 681. Each grantee’s
identifier is a constant.
Treatment/Comparison Group Type (B): Indicates whether the Respondent is receiving treatment or intervention (coded as “1”) or is a control or comparison participant (coded as “2”).
Administrative Format (C): indicates whether this Respondent received an individual/one-to-one intervention at the time of the encounter (coded as “1”) or was involved in a group intervention at the time of the encounter (coded as “2”).
Individual Identifier (D): The Individual Identifier begins with either a Y (assigned to those youth participants under 18 years) or an A (assigned to those adult participants over 18 years) followed by a randomly generated 4-digit number (ex.Y2942). These 5 characters serve as the unique Individual Identifier for each survey Respondent. The unique identifier is assigned by the grantees. The 4 values following the Y or A should be numeric values (not alphanumeric) and only used once per grantee. Numbers can range from 0001 to 9999.
The unique ID number sequence is displayed as follows:
A B C D
---
Example: A survey Respondent might have the following ID number: 601-1-2-A3543. This number tells us that site 601 coded Adult (A) participant 3543 as a member of the treatment group (1) and receiving a group intervention (2) at the time the form was completed.
The data collection anticipated through this study falls under the Federal regulations regarding protection of human subjects in research (45 CFR 46). For all data collection concerning participating youth, informed written consent by parents and assent from the youth will be obtained. The evaluation will involve collecting data from local and re-entry minority youth and adults in communities residing in communities of color. They will complete a self-report questionnaire at three points in time, program entry, program exit, and six months post program exit. Several actions will be taken to protect the identity of the study youth and adults:
All data collected will be maintained in a safe and private manner. The DACCC and grantees will conform to all requirements of the Privacy Act of 1974 under the System of Records, Alcohol, Drug Abuse and Mental Health Epidemiologic Data, HHS/SAMHSA/OA, #09-30-0036.
Grantees will not send identifying information (i.e. name of respondent) to the DACCC. Only a questionnaire identification number will be provided. In addition, grantees will not provide identifying information to CSAP.
Access to the data will be limited to the DACCC staff directly involved in the evaluation. At the end of the grant, a public use data diskette or CD-ROM will be made available containing the HIV program grantees’ findings, along with detailed documentation. These public use data files will contain no individual identifiers. Reports prepared by the DACCC as contract deliverables will present data in aggregate form only.
All DITIC and DACCC staff will take a pledge agreeing that all information provided by respondents will be maintained with complete privacy and security.
All members of the project staff (DITIC and DACCC) will be required to sign a statement of personal commitment (Attachment 1) developed by project staff to guard the privacy of data. Questionnaires will be retained by the contractor for the DACCC for a period of three years and then destroyed, unless otherwise directed by the CSAP Federal Project Officer.
11. Questions of a Sensitive Nature
The questionnaires contain questions of a sensitive nature, such as the use of and attitude towards ATOD because these questions are necessary to obtain data that will help explain observed program outcomes. Since HIV/AIDS can be a sexually transmitted disease, these questionnaires also include a number of questions on sexual orientation, behavior, and practice and their relationship to substance abuse. These questions must be asked because the interventions are focused on the use of safe sexual practices, the use of condoms, and the reduction of unsafe sexual practices due to the contributing influence of ATOD use on acquiring HIV/AIDS and hepatitis C.
Grantees will routinely obtain informed consent from parents of youth participating in the study. Written, informed consent will be a necessary prerequisite at every grantee site prior to data collection. Grantees will guarantee that all data submitted to the contractor for the DACCC has first received the appropriate written consent. This consent will also indicate data collection and release to the DACCC. These consent forms, unique to each grantee, will specify both the risks and benefits of study participation. There are no data elements in the data collection questionnaires covered by the consent that will fall outside of this protection.
,
12. Estimates of the Annualized Hour Burden to Respondents
Table 3 shows the estimated annualized burden for data collection. The evaluation data will be collected through questionnaires administered to youth and adult program participants. Each youth and adult will complete questionnaires three times, taking an average of 50 minutes or 0.83 hours for baseline, exit, and follow-up questionnaires. Approximately 9,000 adults and youths are expected to respond at baseline. It is expected that 6,750 (75%) will respond to the exit questionnaire; and a total of 4,455 (66%) will respond to the follow-up questionnaire. The expected response rates are based on the HIV Cohort 3, 4 and 5 results. The total burden is 16,770 hours and the average annualized burden is 3,354 hours for the five-year program evaluation study. The same numbers are estimated for future cohorts to be funded in FY 2008, 2009 and 2010. The current Cohort 6 estimates are as follows:
Table 3. Estimated Annualized Burden of Data Collection
|
Number of Respondents at Baseline |
Number of Respondents at Exit |
Number of Respond-ents at Follow-up |
Average Burden/ Response (Hrs.) |
Total Burden Hrs. |
Hourly Wage Cost ($) |
Total Wage Cost ($) |
Total of Adults and Youth |
9,000 |
6,750 |
4,455 |
0.83 |
16,770 |
$6.75 |
$113,200 |
The burden estimate presented in Table 3 is based on pilot test experience. There will be no direct cost to youth or adults for participating in the study. The value of youth and adult time was assumed given the prevailing minimum wage rate in California, the State chosen since it is often the leading indicator for setting precedents later adopted by other States.
13. Estimate of Annualized Cost Burden to Respondents
There will be no capital, start up, or operation and maintenance costs incurred by anyone participating in the study. The CSAMS on-line data collection tool will be available at no cost to grantees.
14. Estimate of Annualized Cost to the Federal Government
The total contract award for the DACCC to cover all aspects of the study design, planning, data collection, and analysis is $182,000 over the five year period ($36,400/year). These costs cover the following activities: study design and planning; questionnaire development and pilot testing; assistance to study sites in cooperating with the cross-site evaluation; processing of outcome data from study sites; data analysis and reporting; development of public use data and documentation.
In addition, the total cost for IT activities related to this data collection and data management is estimated to be $180,000 over the five year period ($36,000/year). These activities, provided by the DITIC, include:
The provision of internet-based outcome data submission through CSAMS;
Website and database maintenance, data backup, user account management, and web security;
System update, bug fixing, system enhancement, new module development, and system integration, and
Training and technical support.
Additional costs will be incurred indirectly by the Government in personnel costs of staff involved in oversight of the program evaluation study. It is estimated that one (1) CSAP Project Officer will be involved at approximately 50 percent time, at an average annual salary of $110,000. Direct costs of CSAP Federal staff time will approximate $55,000 annually. The annualized total cost to the Government will be: $36,400+$36,000+ $55,000 = $127,400.
15. Changes in Burden
This is a new data collection.
16. Analysis Plans, Publication Plans, and Time Schedule for the Project
Analysis Plans
The defining characteristic of this cross-site study is that all participating grantees share a common protocol, and a common set of performance measures, outcome objectives, and evaluation questions. This study differs from more traditional multi-site clinical trials because each individual grantee will select an evidence-based practice based on the needs of the particular target population, setting, and organizational characteristics. This multi-site evaluation will not test a single intervention that is implemented in different settings, rather it will test a category of interventions that have similar outcome objectives but that use different approaches to accomplish those objectives.
Analysis of a multiple-site data set requires a complex set of inter-related tasks. Planning for these tasks must be flexible, allowing adjustments as the opportunities and challenges presented by the empirical realities of the data set are discovered. While multi-site studies provide strong opportunities for knowledge generation because of the ability to contrast intervention and implementation variation in a single study, they also present significant challenges. This study recognizes those challenges and anticipated solutions as they will apply to the 81 participating grantees.
Sample Size Determination. Individual grantees have proposed their target population sizes. The establishment of sample size at the grantee level depends to some extent on financial constraints for program intervention/treatment services, staff allocation, and retention activities for data and evaluation activities, including stipends. The overall sample size across all sites is shown in Table 4.
Table 4: Summary of Study Sample Size
Total Population |
Total Population at Entry |
Total Population at Exit |
Total Population at 3-6 Month Follow-up |
Total Adults & Youth |
9,000 |
6,750 |
4,455 |
Statistical Procedure Determination. As a multi-site design, the SAMHSA/CSAP initiative collects information at two levels of observation: 1) across individuals, and 2) within individuals at three points in time. The units of observation also have a hierarchical relation. Individual level units (youth and adult) are nested within program sites and points in time are nested within individuals.
The proposed analysis includes several distinct steps:
First, pooled analyses of outcomes will be conducted to assess the (controlled) presence of significant factors in growth curve trends for youth and adults participating in prevention interventions.
Second, the heterogeneity of outcomes across sites will be assessed to determine if outcomes for SA or important protective factors significantly differ across sites. If there are significant differences, hypotheses will be developed to explain those differences and conduct growth curve analyses on: a) clusters of sites that share characteristics hypothesized to be contributors to effectiveness, and/or b) individual sites that exhibit combinations of principles and practices hypothesized as contributors to effectiveness.
Additional analyses will test the sensitivity of effectiveness models to differences in participant characteristics.
Statistical Test Determinations. Both the structural equation model (SEM) approach to estimating the trajectory parameters and the hierarchical linear model (HLM) approach that can consider time to be nested within an individual will be the key analytic methods conducted for this MSE (Bollen, 1989; Chou, Bender, & Pentz, 1998). This hierarchical data set presents flexible analysis opportunities as well as some analytic challenges. As the primary statistical tool, it is proposed to apply multi-level regression [e.g., SEM and HLM] growth-curve models. This technique allows for the identification of individual effects, controls for co-variates (e.g., propensity scores to control for non-equivalence across intervention groups) and tests for interaction effects with the different types of interventions and youth or adult characteristics that may mediate the impact of the intervention. HLM also provides excellent capacity in analyzing longitudinal, repeated measures designs (Willett, Singer, and Martin, 1998). HLM can also accommodate missing data at individual data points, and allows adjustments for different individual intervals between follow-up data points. Tests for attrition bias and selective attrition are conducted at each follow-up point.
While none of the 80 program sites is incorporating a more robust experimental design with control/comparison groups, the MSE data set will provide the flexibility to conduct analyses that provide useful evidence concerning the general effectiveness of prevention in promoting (developing) significant protective factors that could potentially affect SA or HIV/AIDS or hepatitis risk or transmission. The dataset will also be able to assess the variation in this effectiveness between interventions among the project sites, and offer potential explanations of that variation (e.g., amount of contact, type of intervention approach).
Many of the grantees are targeting African American and Hispanic/Latino populations and populations who have just been released (re-entry) from the criminal justice system. Varieties of methods are being used to recruit participants. Most sites report that participants will be identified from SA treatment programs, referrals from collaborations with criminal justice systems and other involved agencies, and/or through site-specific geographical areas that were identified through the SPF Step 1 needs assessment conducted during the first year of the grant. Key risk factors, including at risk for substance use, HIV/AIDS, or hepatitis transmission, and economically disadvantaged, were the health areas targeted for the needs assessment. For Steps 3 and 4, planning and implementation, grantees are allowed to select and adopt a variety of intervention/treatment approaches to fit the needs of their program participants/clients.
Publication Plans
The MAI cross-site study results will be made available to the public through publications and conference presentations. The following journals carry articles on SA prevention and HIV/AIDS and are expected to serve as potential vehicles for distribution of study results: Journal of Substance Abuse Treatment, International Journal of Addictions, Journal of Community Psychology, Journal of Adolescent Research, Journal of Adolescent Health, Preventive Medicine, Evaluation Review, Policy Studies Review, and the American Journal of Public Health. Study results could also be published in other journals that focus on HIV/AIDS. These include The Journal of the American Sexually Transmitted Disease Association, Health Education and Behavior, AIDS: Official Journal of the International AIDS Association, AIDS Education and Prevention, The Journal of Sex Research, AIDS Care, Psychological and Socio-Medical Aspects of AIDS/HIV, and Current Opinion in HIV and AIDS. Study results also are targeted for publication in journals focusing on infectious diseases. These include, among others, The Journal of the American Microbiological Association and the Journal of Infectious Diseases.
The study results will be distributed through presentations at annual conferences of national and international public health organizations such as the Society for Prevention Research, the American Public Health Association, the National Association of Alcohol and Drug Abuse Counselors, The National Prevention Network, the American Evaluation Association, and HIV/AIDS national meetings as well as regional and State SA prevention and treatment associations. HIV/AIDS meetings could include, among others, CDC Annual Conferences on AIDS and Conferences of the International AIDS Society. Results could also be presented at meetings focusing on infectious diseases such as annual meetings of the American Society of Microbiology.
Documents will also be prepared and published on behalf of the Government (CSAP/SAMHSA) through the Government Printing Office (GPO) for Federal agency and public use. Findings will also be available via OMB’s Website: www.expectmore.gov, as well as in annual Reports to Congress and the performance detail sections of annual SAMHSA budgets as they become publicly available.
Timeline
The MAI is a five-year grant program (see Table 5). Years I and 2 are devoted to Steps 1, 2 and 3 of the SPF, namely conducting the needs assessment, capacity building, and planning, respectively. Years 3, 4 and 5 are devoted to Steps 4 and 5, implementation and evaluation, respectively.
Table 5. Project Timeline.
Date |
Activity |
|
|
FY2007 |
Needs Assessments, Capacity Building, Planning |
FY2008 |
OMB Clearance Obtained, Implementation |
FY2009 |
Implementation |
FY2010 |
Analysis and Reporting |
17. Display of Expiration Date
The expiration date will be displayed on all survey forms.
18. Exceptions to Certification Statement
The certifications are included in this submission.
REFERENCES
Bentler, P.M., EQS Structural Equations Program Manual.
Encino, CA: Multivariate Software, Inc., 1995.
Bollen, K.A., Structural Equations with Latent Variables. New
York: Wiley, 1989.
Centers for Disease Control and Prevention. Coinfection with HIV and Hepatitis C Virus. CDC HIV/AIDS Fact Sheet. Atlanta, GA: Centers for Disease Control and Prevention, 2005. From: http://www.cdc.gov/hiv/resources/Factsheets/PDF/coinfection.pdf (Accessed July 11, 2008).
Centers for Disease Control and Prevention. Factsheets. HIV/AIDS Atlanta, GA: Centers for Disease Control and Prevention, 2008. From: http://www.cdc.gov/hiv/resources/factsheets/index.htm (Accessed July 14, 2008).
Chou, C.P., Bentler, P.M., Pentz, M.A., Comparison of two statistical approaches to the study of growth curves: The multilevel model and latent curve analysis. Structural Equation Modeling, 5, 247-266, 1998.
Herrell, James M. and Straw, Roger B. (Eds.) New Directions for Evaluation: Conducting Multiple Site Evaluations in Real-World Settings. Hoboken, NJ: Jossey-Bass, 2002.
Koziel, M.J., and Peters, M.C., Viral hepatitis in HIV infection, New England Journal of Medicine, 14:356,1445-1454, 2007.
Office of Applied Studies. National Survey of Substance Abuse Treatment Services (N-SSATS): 2005. Data on Substance Abuse Treatment Facilities. DASIS Series: S-34 (Rep. No. DHHS Publication No. (SMA) 06-4206). Rockville, MD: Substance Abuse and Mental Health Services Administration, 2006.
Springer, F., "Lessons learned from multiple cross-site evaluation studies.” Paper presented at the American Evaluation Association Meeting, Honolulu, 2000.
Summary Fact Sheet on HIV/AIDS: The HIV/AIDS Epidemic: 20 Years in the U.S. The White House, Washington, D.C. 2008. From: http://www.whitehouse.gov/onap/facts.html (Accessed July 14, 2008).
Tedaldi, E. M., Hullsiek, K. H., Malvestutto, C. D., Arduino, R. C., Fisher, E. J., Gaglio, P. J. et al., Prevalence and characteristics of hepatitis C virus coinfection in a human immunodeficiency virus clinical trials group: The Terry Beirn Community Programs for Clinical Research on AIDS. Clinical Infectious Diseases, 36, 1313-1317, 2003.
Willett, J.B., Singer, J.D., & Martin, N.C., The design and analysis of longitudinal studies of development and psychopathology in context: Statistical models and methodological recommendations. Development and Psychopathology, 10, 395-426, 1998.
Winstock, A.R., Anderson, C.M., and Sheridan, J., National survey of HIV and hepatitis testing and vaccination services provided by alcohol agencies. Medical Journal of Australia, 184, 560-562, 2006.
B. Collection of Information Employing Statistical Methods
1. Respondent Universe and Sampling Methods
The plan is to assess the universe and based on this, grantees will propose their target population sizes. The establishment of the sample size at the grantee level depends to some extent on the size of their client population, the financial constraints on the program intervention/treatment services, staff allocation and retention, and the availability of stipends for data collection activities. Table 4 displays the aggregate baseline sample sizes for program participant groups. As noted, the expected response rates at exit are 75% and 66% at 3-6 month follow-up are based on the HIV Cohort 3, 4 and 5 results.
2. Information Collection Procedures
Common Measures - Youth and Adult Questionnaires (Completed by Program Clients)
The study uses a common protocol for collecting program and participant/client level data and submitting it to SAMHSA via an online, Web-based data entry system (CSAMS). The DITIC will assist grantees with online data entry. The protocols are described below:
Youth and Adult Outcome Questionnaires. Two questionnaires will be administered to program participants/clients. Each questionnaire will be administered three times, as follows:
Youth Questionnaire: For persons aged between 12 and 17 at baseline, exit, and follow-up
Adult Questionnaire: For persons aged 18 and older at baseline, exit, and follow-up
Both the Youth and Adult survey forms include the NOMs measures (OMB No. 0930-0230) plus a set of measures specific to HIV prevention. For all common measures, administration guides have been prepared to assist program sites with implementation.
While the GPRA and NOMs measures have already been approved by OMB (OMB No. 0930-0230), the remaining HIV-related constructs have not, and require OMB approval. The questionnaires contain 135 questions, of which 102 relate to sexual behavior, knowledge of HIV and hepatitis, and health care. Questions for both the adult and youth questionnaires do not change across the three data collection points (e.g., baseline, exit, and follow-up) (See Attachments 2 and 3 for a copy of the Youth Baseline Questionnaire and the Administration Guide, and Attachments 4 and 5 for a copy of the Adult Baseline Questionnaire and the Administration Guide). The planned analysis will assess the degree to which program strategies reduce SA/HIV/AIDS and hepatitis risk and increase SA/HIV/AIDS and hepatitis protective factors among minority and minority re-entry populations.
The information obtained from these questionnaires will generate data to determine the effectiveness of the program in reducing SA, IDU, and high-risk sexual behaviors as well as increasing participant knowledge. Pilot tests have been conducted to assist in controlling respondent burden and ensuring the user-relevance of questions.
Data Collection
Time Points (Efficacy) for Data Administration: The common design includes assessments at baseline, program exit, and three to six months post-exit (follow-up). The common questionnaires will be administered to all intervention (program participants) youth and adults at baseline (first data collection point), program exit (second data collection point), and follow-up (third data collection point).
Baseline Questionnaires: Should be administered within 30 days of intake or before core program services begin and are considered to be the first data collection point.
Exit Questionnaires: Should be administered again within 10 days post program exit or after core program services have ceased and are considered to be the second data collection point.
Follow-up Questionnaires: Should be administered within 30 days of the planned follow-up (post-exit) and are considered to be the third data collection point. Most study sites have a planned follow-up administration of three to six months.
The common questionnaire assessment schedule is depicted in Table 6.
Table 6. Common Instrumentation Assessment Schedule
Questionnaire |
Baseline |
Exit |
Follow-up |
Youth Questionnaire |
X |
X |
X |
Adult Questionnaire |
X |
X |
X |
This time series analysis is depicted graphically below:
01 X 02 03
Where: 01= Baseline - First Data Collection Point
X = Program Prevention Intervention Activities
02 = Exit - Second Data Collection Point
03 = Follow-Up – Third Data Collection Point
Sample size, respondent burden, and intrusiveness have been minimized to be consistent with the cross-site analytical objectives. Procedures are employed to safeguard the privacy and security of participants’ responses. Every effort has been made to coordinate cross-site data collection with local data collection efforts in order to minimize respondent burden. The pilot study results indicate that the questionnaires require approximately 50 minutes or 0.83 hours to administer.
Data Management.
Storage System: Over the life of this initiative, each grantee will be collecting information that must be documented and organized. Each local evaluation team or the person responsible for data management will be required to store:
Individual questionnaires (Youth & Adult and Individual & Group Dosage forms) until they are entered or uploaded into CSAMS,
Consent forms, and
Tracking forms for each of the participants in the study.
Before initiating data collection, each site is responsible for setting up a filing and storage system that will accommodate these needs.
Web-Based Data Entry Upload System: The DACCC has created CSAP’s Services Accountability & Monitoring System (CSAMS), an online data entry system that provides prevention information, data collection tools, documents, data entry functions, and access to reporting statistics and tracking. All of the HIV questionnaires can be found in the “Tools” section of this Website. Common questionnaires are available in both Microsoft Word and PDF format for individual grant sites to download and make copies for administration to clients or participants. Site evaluators or data collectors are expected to enter client or participant responses to questionnaires through the CSAMS Website. Sites will also be able to upload response databases through CSAMS that use the appropriate variable/value numbering (Questionnaire codebooks are also available on the “Tools” section of CSAMS Website). Dosage data will also be entered online via the CSAMS Website by the Project Director or Evaluator of the grant site.
Once data have been entered into CSAMS, the DACCC will clean the data and each grant site will have access to their downloadable, clean, electronic data files. Once data are entered by the grant site and cleaned by the DACCC, data will be available for download by the grant site for use in local data analysis and reporting. Sites are able to enter the data online, or upload if necessary, on a continual basis. The DACCC will abstract data biannually for cleaning, analysis, and reporting purposes; however the data will remain accessible for local evaluations.
CSAP has established a Help Desk through the DITIC contract to assist program sites with data collection and data entry. Assistance is available by calling the Help Desk at: 1-888-DITIC-4-U (348-4248), or email: [email protected]
3. Methods to Maximize Response Rates
The expected response rate at exit is 75% and at three to six month follow-up is 66%. These estimates are based on the HIV Cohort 3, 4, and 5 results. As noted above, each grantee has developed a package of incentives to maximize their response rates. These incentives have been selected on the basis of each grantee’s past experience with their client population. Typically, prevention programs have lower retention rates than treatment programs so these estimates fall below the OMB expected norm of 80%. In order to achieve these retention rates, it is also necessary to provide incentives.
4. Tests of Procedures
CSAP and its contractor have reviewed the adult and youth questionnaires, clarifying terminology and language, and rewriting or eliminating questions that were unclear or unnecessary. Questionnaires were then pilot-tested on a small sample of less than 10 individuals to ensure that the multi-site assessment requirements and procedures were consistent with activities conducted at sites. The pilot-testing was designed to determine what questions should be included, how should these questions be asked, and how participants would be expected to respond. In order to obtain this information, three types of assessments were conducted, including participants’ written and oral feedback, testing administrators’ written and oral feedback, and pilot test data. The pilot test data included data distribution and patterns, missing data, and data psychometric properties. The administration time was an average of 45 minutes for the youth questionnaire and 50 minutes for the adult questionnaire. As a result of these assessments, some items were reworded, some items were added, some items were dropped, and some response formats were modified. However, the small number of participants (N) precluded a thorough reliability analysis of the psychometric properties of the study scales. Despite this low N, the pilot test process helped strengthen the final study instruments. As a result, study items were changed/modified for easier questionnaire administration, clearer meaning of the questions to the participants, and more robust analyses of study data.
The purpose of the pilot testing was to:
Identify instructions and/or questions within the questionnaire that were unclear or confusing
Obtain suggestions for improving questions or instructions
Identify data collection procedures that were unclear or confusing
Measure respondent burden based on the average time to complete questionnaires
Identify ways to minimize respondent burden and improve accuracy in the completion of questionnaires
Comments on the draft feedback forms included collecting information on the likelihood of obtaining specific responses, overall questionnaire layout, item flow, and administration.
5. Statistical Consultants
Virginia Mulkern, Ph.D.
Senior Vice President
Human Services Research Institute
2336 Massachusetts Avenue
Cambridge, MA 02140
Nilufer Isvan, Ph.D.
Senior Research Fellow
Human Services Research Institute
2336 Massachusetts Avenue
Cambridge, MA 02140
P.
Allison Minugh, Ph.D.
President/CEO
Datacorp, Inc.
Two
Richmond Square, Suite 100A
Providence, Rhode Island 02906
Attachments
HIV/AIDS DACCC/DITIC Statements of Personal Commitment
HIV/AIDS Youth Questionnaire
HIV/AIDS Youth Questionnaire Administration Guide
HIV/AIDS Adult Questionnaire
HIV/AIDS Adult Questionnaire Administration Guide
| File Type | application/msword |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |