HIVOMBSupportingStatement090318

Minority AIDS Initiative

OMB Supporting Statement

A. JUSTIFICATION

1. Circumstances of Information Collection

The purpose of this request is to obtain clearance to conduct a cross-site study of the Minority HIV/AIDS Initiative (MAI) funded by the Substance Abuse and Mental Health Services Administration (SAMHSA)’s Center for Substance Abuse Prevention (CSAP) from September 2005 to September 2010. The primary objectives of the cross-site study are to:

• Assess the process of adopting and implementing the Strategic Prevention Framework (SPF) with the target populations.

• Measure the effectiveness of evidence-based programs and infrastructure development activities such as: outreach and training, mobilization of key stakeholders, substance abuse and HIV/AIDS counseling and education, referrals to appropriate medical treatment and/or other intervention strategies (i.e., cultural enrichment activities, educational and vocational resources, and computer-based curricula).

• Determine the success of the MAI in preventing, delaying, and/or reducing the use of alcohol, tobacco, and other drugs (ATOD) among the target populations. The results of this cross-site study will assist SAMHSA/CSAP in promoting and disseminating optimally effective prevention programs.

This program is authorized by Section 516 of the Public Health Service Act, as amended, and subject to the availability of funds. It was supported by the Congressional Black Caucus through its Conference Report on H.R. 4328, Making Omnibus Consolidated and Emergency Supplemental Appropriations Act, for FY 1998 (House of Representatives, October 19, 1998), to address prevention and treatment needs of minority communities that are heavily affected by HIV/AIDS and hepatitis. It builds on previously authorized programs addressing these issues (discussed below). In addition, the program addresses the DHHS’s Healthy People 2010 goals and objectives 26-1, 26-3, 26-7, 13-7, 13-8, and 25-12, the 20 Department–Wide Objectives regarding substance abuse prevention, and SAMHSA’s HIV Matrix priorities.

Although several Federal agencies have mandates to fund projects targeting minority and minority re-entry populations who are at risk for substance use, HIV/AIDS, and hepatitis, very little is known about the efficacy of such programs once they become widely disseminated. Prior efforts to evaluate government substance use prevention initiatives targeting at-risk populations have focused on highly specific program models and strictly defined target groups, or have been hampered by lack of valid instrumentation and poor study design. As models were disseminated to community-based agencies (that typically implement these programs under less rigorous and controlled parameters), measures and efforts to assess outcomes were inadequate and/or not sufficiently designed to determine the true impact of these interventions. In addition, the link between substance abuse and HIV/AIDS and hepatitis outcomes has not been evaluated for these programs or in local community settings.

Literature searches within SAMHSA’s three centers and with five other Federal agencies were conducted to identify studies with similar goals and expected outcomes. No similar study has been conducted that examines prevention initiatives regarding substance abuse (SA) in relation to HIV/AIDS and hepatitis. Even though the present cross-site study is unique from others that have been conducted in the field, information generated from these literature searches has sharpened the present cross-site study design and enhanced the likely utility of the results.

Data will be collected from approximately 35,300 respondents (88% adults; 12% youth) served by the 81 grantees at three time points: baseline (program entry), program exit, and three to six months post-exit. The CSAP National Outcomes Measures (NOMs) Adult and Youth questionnaires, which have been approved by OMB (OMB # 0930-0230) for use in all CSAP evaluation studies, will be used to measure ATOD use and risk factors associated with ATOD use among program participants. These NOMs data are used to report on Government Performance and Results Act (GPRA) and Performance Assessment Rating Tool (PART) findings across CSAP programs. For this program, these cross-site instruments are augmented with additional scales to measure other important risk and protective factors uniquely associated with HIV/AIDS and hepatitis among minority populations and minority re-entry populations in communities of color. Each questionnaire contains 135 questions, of which 102 relate to HIV/AIDS and hepatitis. Grantees will report these data online through the CSAP Services Accountability Monitoring System (CSAMS).

The cross-site study is scientifically appropriate, employs measures to safeguard the privacy and security of participants’ responses, and supports the program and study needs of multiple Federal agencies. Sample size, respondent burden, and intrusiveness have been minimized to be consistent with cross-site study objectives. To minimize and control respondent burden and to ensure the user-relevance of questions, every effort has been made to coordinate cross-site data collection with local data collection efforts, including pilot testing.

The cross-site study results will have significant implications for the substance abuse, HIV/AIDS and hepatitis prevention field, the allocation of grant funds, and other evaluation activities conducted by multiple Federal, State, and local government agencies. The results will be used to develop Federal policy in support of SAMHSA/CSAP program initiatives, inform the public of lessons learned and findings, improve existing programs, and promote replication and dissemination of effective prevention strategies.

Background

SAMHSA is the Federal agency with lead responsibility for the prevention and treatment of addictive problems and disorders. As part of its mission, the Agency acts to:

• Reduce the incidence and prevalence of substance abuse, and improve treatment outcomes for persons suffering from addictive disorders; and

• Improve access and reduce barriers to high quality, effective prevention and treatment programs for drug abusers.

As one of SAMHSA’s three centers, CSAP serves as SAMHSA’s focal point for substance abuse prevention initiatives among youth and other high-risk populations as mandated by Section 517 of the Public Health Service (PHS) Act (42 USC 290bb-23).

Epidemiological studies on the dynamics of substance abuse, HIV/AIDS, and hepatitis demonstrate a continued need to reach out to communities of color, particularly to those reporting high rates of HIV/AIDS, hepatitis, and sexually transmitted diseases (STDs). The Centers for Disease Control and Prevention (CDC)’s Morbidity and Mortality Weekly Report indicates that approximately 37,000 persons were diagnosed with HIV/AIDS in 2005, of which 49% were Black (not Hispanic), 18% Hispanic, 1% Asian-Pacifier Islanders, 1% American Indian and Alaskan Native, and 3% White (CDC: MMWR, November 18, 2005, 1153). CDC also reported that more than one-third of all HIV infections progressed to AIDS within 12 months after diagnosis. In 2005, CDC reported injection drug use (IDU) as the exposure category for 14 % of male and 19% of female new AIDS cases. Due to similar transmission routes, many minority populations at risk for HIV/AIDS are also at risk for viral hepatitis. For both HIV/AIDS and viral hepatitis, IDU individuals are an important target population. Between 50% and 90% of persons infected with the hepatitis C virus (HCV) report IDU as the risk (CDC: Co-Infection with HIV and Hepatitis C Virus, November 2005).

Of particular concern to communities of color is the return of ex-offenders, otherwise known as the re-entry population. Despite the efforts of correctional facilities to prevent sexual risk-taking behavior and substance abuse among incarcerated persons, a significant number engage in high-risk activities (such as IDU, tattooing, and coerced sexual activity), placing others at risk for HIV and hepatitis transmission. Each year, many of these persons, unaware of their HIV and hepatitis status, return to their communities and re-engage in substance abuse and other high-risk behaviors, putting others at an even greater risk for HIV/AIDS and hepatitis transmission.

No matter how they are transmitted, HIV/AIDS and viral hepatitis are infectious diseases that have drastic long-term medical, economic, and social consequences on minority populations. Meeting the challenges posed by viral hepatitis requires close coordination with existing local, State, and territorial substance abuse, HIV/AIDS, and hepatitis prevention programs. SAMHSA is working to improve access to quality services by increasing outreach and service capacity to at risk populations of color. Use of the SPF as a method to prevent and reduce both substance abuse and the transmission of HIV/AIDS and hepatitis will lay the necessary foundation for effective and sustainable prevention service delivery in the context of substance abuse and HIV/AIDS.

In FY 2005, the current MAI Cohort 6 Program funded 81 five-year grants to community-based organizations. This program combines planning and services funding and requires all grantees to participate in this cross-site study. They are expected to provide leadership and coordination on the planning and implementation of the SPF that targets minority populations and the minority re-entry population in communities of color with a high prevalence of substance abuse, HIV/AIDS, and hepatitis. The primary objectives of the cross-site study are to:

• Assess the process of adopting and implementing the SPF with the target population,

• Measure the effectiveness of specified intervention strategies such as cultural enrichment, educational and evaluation activities, vocational resources, and/or computer-based curricula, and

• Determine the success of the MAI in delaying, preventing, and/or reducing the use of alcohol, tobacco, and other drugs among the target population.

This MAI Cohort 6 program builds on five previous SAMHSA/CSAP HIV/AIDS grant programs that provided substance abuse and HIV/AIDS planning and prevention services for minority populations, including:

• HIV Cohort 1 Services Grants – 48 funded in FY 1999 for 3 years

• HIV Cohort 2 grants funded in 2001:

  1. Planning Grants – 41 funded for 1 year

  2. Expansion Grants – 20 funded for 3 years

  3. Youth Services Cooperative Agreements – 18 funded for 3 years

• HIV Cohort 3 Targeted Capacity Expansion Initiative for SA and HIV Prevention in Minority Communities Grants funded in 2001:

  1. Planning Grants – 10 funded for 1 year

  2. Services Grants – 48 funded for 3 years

• HIV Cohort 4- Targeted Capacity Expansion Initiative for SA and HIV Prevention in Minority Communities Grants funded in FY 2003:

  1. Planning Grants – 10 funded for 1 year

  2. Services Grants – 22 funded for 5 years

• HIV Cohort 5 Services Targeted Capacity Expansion Initiative for SA and HIV Prevention in Minority Communities Grants - 46 grants funded in FY 2004 for 4 years

As noted, the goal for Cohort 1 grants was to provide services. Goals for Cohorts 2-5 grants were to add, increase, or enhance integrated substance abuse and HIV prevention services by providing supportive services and by strengthening linkages between service providers for at-risk minority populations. HIV Cohorts 1-3 grants were funded under SAMHSA/CSAP’s umbrella OMB Clearance Document 0930—0208. Since neither the HIV Cohort 4 nor the Cohort 5 Programs were cross-site studies, they did not require OMB clearance.

These past programs have enabled CSAP to make great progress in providing innovative, community-based drug prevention, planning, and intervention services to minority populations at risk for substance abuse and HIV/AIDS. Cohort 3 results show reductions in ATOD use and in risky sexual behavior, and an increase in education pertaining to HIV/AIDS and hepatitis prevention. Between baseline (n=80) and exit (n=131), there was a significant decrease in the number of participating youth who reported having sex after getting drunk or high, and a significantly higher number of participants taking HIV/AIDS education classes at exit (n=2,737) than at baseline (n=1,237). Among Cohort 3 adult participants, the number smoking cigarettes decreased from baseline (n=194) to exit (n=176), and the number drinking alcohol decreased from baseline (n=85) to exit (n=59).

The Cohort 6 MAI Program differs substantially from the earlier programs in that it targets a very different population and calls for the use of the SPF and evidence-based programs. While these grantees have substantial flexibility in choosing evidence-based programs, they are all required to base their projects on the five steps of SAMHSA’s SPF to build service capacity specific to substance abuse, HIV/AIDS, and hepatitis prevention (HP) services. In FY 2006, all the grantees initiated Steps 1, 2, and 3 of the SPF, namely conducting a Needs Assessment, Building Capacity, and Planning. Once their plans are approved by their Project Officers, they can proceed to Step 4 - Implementation, and then to Step 5 - Evaluation.

2. Purpose and Use of Information

CSAP is seeking approval from OMB to expand the NOMs youth and adult questionnaires (OMB No. 0930-0230) by adding items unique to the MAI program and necessary to conduct a systematic cross-site analysis of the intervention and/or infrastructure development services provided by grantees supported by this program. This cross-site study will involve not only collecting information on the planning and delivery of the evidence based programs, but also assessing their effectiveness. Grantees will be conducting ongoing monitoring and analysis of their projects to assess program effectiveness, including Federal reporting of the 1993 GPRA, PART, and SAMHSA/CSAP NOMs, as well as HIV Counseling and Testing.

CSAP wishes to continue to enhance the nation’s impact on the HIV/AIDS epidemic with the current cross-site study, as data collection and analysis is designed to advance the current state of knowledge about the effectiveness of prevention programs for minority populations at risk for SA, HIV/AIDS, and hepatitis, and to provide evidence and conclusions for disseminating optimally effective prevention policy and programs. Information collected will be used by CSAP, SAMHSA, and other Federal agencies in their efforts to assess specific intervention services in the prevention or reduction of substance use, HIV/AIDS, and hepatitis among minority community and re-entry populations in communities of color across the nation.

Information will also be useful to policymakers, who need to learn how to extend their reach into and among these populations.

CSAP will also share the outcome information and lessons learned with other Federal DHHS agencies, including but not limited to SAMHSA’s Center for Substance Abuse Treatment (CSAT), the National Institute on Drug Abuse (NIDA), the Centers for Disease Control and Prevention (CDC), and the Administration for Children and Families (ACF), which administers several drug-related programs targeted at hard-to-reach and at-risk populations.

Beyond DHHS, CSAP plans to share outcomes and lessons learned with:

• The Department of Justice (DOJ) and their Office of Juvenile Justice and Delinquency Prevention (OJJDP), which funds projects that target high-risk youth and often involve SA prevention interventions.

• The Department of Housing and Urban Development (HUD), which supports two major initiatives (the National Youth Sports Program and the Public Housing Drug Elimination Program) that target youth at risk of substance use and provide positive alternative activities for at-risk youth in a drug-free environment.

• The Department of Education (DOE), which funds the Safe Schools/Healthy Students effort (focused on violence and substance abuse prevention) under the Drug Free Schools and Communities Act.

• State and local program planners and the public through publications and a public-use data set.

Implementing evidence-based programs in minority community settings presents challenges (i.e., maintaining rigor in design and instrumentation, as well as maintaining the ability to measure impact), given the need for local adaptations for specific target populations. Using the lessons learned from the previous programs, the current focus is on enhancing the effectiveness of specific interventions for reducing risk factors and/or enhancing the factors that protect against ATOD abuse and HIV/AIDS among the minority youth and adult population as well as the minority youth and adult re-entry population.

CSAP has a well-established history of incorporating evaluation findings and conclusions into the policy process, and the results of this study will be similarly used. As it is designed to provide more specific information on the effectiveness of diversely funded programs in preventing and/or reducing ATOD use and related problems, CSAP will use the evaluation results to influence public policy, research, and programming as they relate to the provision of youth and adult services. More specifically, the research will support the following uses by CSAP:

• Findings will be used in required NOMs, GPRA, and PART performance reporting, and will be presented in annual reports to Congress.

• Findings regarding SPF implementation will be used to delay or reduce ATOD use, influence positive sexual behaviors, change ATOD attitudes, and reduce associated problem sexual and substance use behaviors in participants, and as well as to assess the effectiveness of currently funded prevention programs. Furthermore, the common use of ATOD outcome measures (from CSAP, GPRA, PART, and NOMs) will allow CSAP to compare initiatives (including the previous HIV/AIDS programs). Such extensive cross-initiative information will be used to set broad prevention policy priorities.

• Findings concerning the ATOD and sexual behavior risk factors as both program outcomes and mediating factors will be used to refine policy and shape future program funding announcements. In addition, the findings may be used to provide recommendations to States regarding selection of evidence-based programs, since a portion of Block Grant monies given to the States must be spent on SA prevention. Additional monies have been awarded to some States through the Strategic Prevention Framework State Incentive Grants (SPF-SIG).

• Findings concerning program inputs (intervention strategies, frequency, and length) will be used to provide program guidelines (e.g., through RFA’s) and to plan appropriate technical assistance services for programs/States.

• Findings will support CSAP publications and materials on prevention practices that are an important resource for public and private organizations involved in the design and implementation of prevention programming for youth and adults.

In sum, the findings from the study will be a crucial resource for CSAP in setting prevention policy priorities, measuring performance, and designing and promoting optimally effective prevention program initiatives. Although the study is designed primarily to address CSAP program requirements, the cross-site study results will be useful to other Federal, State, and community agencies involved in efforts to prevent or reduce ATOD use among youth and adults. While some of these agencies are specifically interested in providing preventive health services, others have a more general interest in approaches or strategies that have been proven effective. The following are illustrative of the potential uses of data from the MAI cross-site study by Federal organizations and agencies outside of CSAP:

• The Center for Substance Abuse Treatment (CSAT) may use the data and findings to design youth and adult treatment programs that address those risk factors most strongly related to ATOD abuse and HIV/AIDS.

• The National Institute of Allergy and Infectious Disease (NIAID) may use findings to inform and refine their own basic research programs in prevention of HIV/AIDS and hepatitis. The MAI cross-site study will identify particularly promising intervention strategies addressing risk, resiliency, ATOD use, and HIV/AIDS. Study findings will be useful for targeting future controlled studies of prevention interventions.

• The National Institute on Drug Abuse (NIDA) may use findings to inform and refine its own basic research programs in prevention. The MAI cross-site study will identify particularly promising intervention strategies addressing risk, resiliency, ATOD use, and HIV/AIDS. These study findings will be useful for targeting future controlled studies of prevention interventions.

• The Administration for Children and Families (ACF) may use the study findings to guide its research and programmatic efforts involving children and youth. The findings will also help the Agency in its planning of future services funded under The Drug Abuse Prevention Program for Runaway and Homeless Youth and The Youth Gang Prevention Program.

• The Department of Education (DOE) may use study findings to provide local educational agencies with information necessary to enhance the success and effectiveness of after-school programs for youth.

• The Department of Housing and Urban Development (HUD) may use the MAI cross-site findings to improve the program designs of both the Public Housing Drug Elimination and Youth Sports Programs. More importantly, HUD has an interagency agreement with DOJ to conduct ATOD prevention training for public housing authorities that use the risk and resiliency factor model.

• The Department of Justice (DOJ) may use the study findings to guide its program efforts to prevent and treat HIV/AIDS transmission among prison inmates and in serving its juvenile justice population.

State and local agencies also have significant responsibilities for design and implementation of prevention programs for youth and adults. The results of the MAI cross-site findings may be useful in a variety of ways to State and local agencies, including:

• Policymakers in State and local governments will have evidence of the impact of various evidence-based programs and infrastructure development models on preventing or reducing ATOD use and HIV/AIDS and hepatitis among minority and re-entry youth and adults residing in communities of color. The evidence will be useful in setting prevention policy priorities.

• Program planners in State and local governments and in community-based organizations will have comparative evidence on the effectiveness of different models for the provision of youth and adult services. This information will be useful in developing funding guidelines and direct service programs.

• National, not-for-profit, nonprofit, voluntary, and professional organizations will have an accurate portrayal of the program inputs that are required in successful programs targeting minority and re-entry youth and adults residing in communities of color. This information will promote optimally effective prevention program design.

3. Use of Information Technology

It is anticipated that technical infrastructure and data management skills will vary across grantee sites. To maximize data accuracy and reliability, online data entry tools will be designed for the instruments being submitted for clearance. These tools will be made available to grantees through CSAP’s Services Accountability and Monitoring System Web portal (CSAMS). The tools will be designed to reflect the structure of the instruments, and to allow the entry of data from completed survey forms directly into the system through the use of radio buttons corresponding to response options. The system will automatically quantify the selected response options and store the numeric codes in a SQL server for subsequent extraction, cleaning, and analysis.

CSAMS is maintained by CSAP’s Data Information Technology Infrastructure Center (DITIC). The data entered online by grantees are periodically extracted by DITIC and transmitted in encrypted form to CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) for cleaning, record linkage, and analysis. Grantees have two options for accessing the data they entered online. In the first option, grantees can download the raw data they have entered online (as soon as it is submitted) in spreadsheet form. They can also access their data from the cleaned analysis files prepared by DACCC posted on CSAMS under password protection. In the second option, grantees can upload complete data files to CSAMS. For this option, grantees are required to use a standard codebook while preparing the data, thus ensuring that uploaded data files have the same numeric coding and variable naming conventions as the data entered using the online tools.

These online data entry tools reduce the grantees’ burden by facilitating the data entry process and minimizing coding and variable naming errors. They also allow grantees without access to data management/analysis software to accurately quantify the information in completed survey forms. The DACCC will then conduct cross-site analyses to determine outcomes for the program as a whole.

The proposed electronic multi-site data collection process will increase the efficiency and practical utility of the assessment of these programs. The CSAP multi-site questionnaires have been developed and used by grantees in previous HIV cohort programs and have been demonstrated to work efficiently and effectively. Based on the feedback of the HIV pilot, the questionnaires and procedures for electronic transmission of data files have been improved to increase efficiency and minimize burden on both training participants and grantee staff.

4. Efforts to Identify Duplication

CSAP conducted an extensive literature search, consulted with staff in Federal agencies and organizations that work with ATOD and HIV/AIDS and hepatitis prevention programs, and discussed the proposed program with substance abuse prevention experts. Specifically, CSAP:

• Conducted a comprehensive literature search of completed and ongoing studies of ATOD and HIV/AIDS prevention programs targeting youth and adults and found insignificant duplication with this cross-site study. All studies were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or application of results. The search found that there has been no outcome evaluation of substance prevention programs of comparable scope to this study — no study spanning substance abuse, HIV/AIDS, and hepatitis for minority youth and minority re-entry youth residing in communities of color — and no study has used the same measures as those proposed to document outcomes across youth and adults. As mentioned previously, evaluations of previous CSAP HIV Cohorts have been conducted. These past program designs suffered from severe methodological flaws that limited the ability to detect outcomes and produce meaningful results. The present initiative seeks to improve upon the lessons learned by applying more rigorous analytical procedures and a more tailored design.

• Consulted with staff in CSAT, CDC, NIAID, NIDA, ACF, OJJDP, HUD, DOE and DOJ. None of these Federal organizations has conducted a cross-site outcome evaluation of prevention and early intervention programs targeting minority youth and minority re-entry youth residing in communities of color that is similar to the one being proposed for this study.

• Staff attended national meetings at which completed, ongoing, or contemplated evaluations were discussed and found insignificant duplication with the proposed study.

In summary, CSAP did not identify any redundancy in that there were no precedents for a cross-site study of projects like the one being proposed. The studies that they did find were limited to general populations, HIV/AIDS alone, or HIV/AIDS and substance abuse but not hepatitis. Others suffered severe methodological problems such as low sample sizes, lack of dosage monitoring, and/or lack of published/reliable/valid measures and scales that make it unlikely that current information will be published or released among the scientific community or in respected journals, government publications, etc. Thus, it is clear that the data to be collected will be unique to the CSAP MAI programs, collected only for the CSAP programs, and not available elsewhere. The data collected through the multi-site effort will be non-duplicative, minimize burden on respondents, and be of use to both CSAP and the communities of color.

5. Involvement of Small Entities

This data collection will have no significant impact on small entities.

6. Consequences if Information is Collected Less Frequently

The data will be collected from participants at three points in time: baseline (program entry), program exit, and three to six months post-exit. Failure to collect the information from all participants at all three points in time will result in missed opportunities and lessons learned on how to provide a quality improvement mechanism for CSAP to continually monitor and refine its prevention programs to ensure they meet the needs of minority populations and minority re entry populations at risk for SA/HIV/AIDS and hepatitis residing in communities of color. Data collected at all three points in time is essential, as it will also demonstrate whether sustainable results can be achieved over time after the program has ended, and if so, for which types of interventions and populations.

Without this information:

• CSAP will not be able to determine the extent to which it can prevent, reduce, and/or delay substance abuse and, in turn, reduce other risky behaviors that can lead to HIV/AIDS and hepatitis infection among minority populations and minority re-entry populations residing in communities of color.

• CSAP will not be able to monitor the quality of its prevention programs and determine how they can be improved to ensure continued success at meeting the needs of minority populations at risk for HIV/AIDS and hepatitis.

• CSAP will not be able to describe fully the range of prevention services used and the efficacy of evidence-based programs.

• CSAP will not be able to ascertain if participants are more knowledgeable about HIV/AIDS and hepatitis and how they relate to SA as a result of program participation.

• CSAP will not be able to identify those prevention services that are most effective and identify the potentially unique needs of minority populations residing in the community and minority re-entry populations.

• CSAP will not be able to meet its Federal reporting requirements to DHHS, OMB, and Congress.

7. Consistency With the Guidelines in 5 CFR 1320.5(d) (2)

This information collection fully complies with 5 CFR 1320.5(d) (2).

8. Consultation Outside the Agency

a. Federal Register Notice

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on Wednesday, August 22, 2007, on pages 47055-47056. No comments were received.

CSAP has consulted experts both within and outside of the Agency on refinement of the design, instrumentation, products, and statistical aspects of the cross-site study at critical junctures during the study design. These consultations provided an opportunity to: obtain advice and recommendations on the identification and prioritization of the information to be gathered; to ensure the technical quality, appropriateness, and user relevance of the study results; to verify the importance, relevance, and accessibility of the information to be sought; and to minimize respondent burden.

b. Consultations Within the Agency:

The multi-site study and questionnaire design were based on initial consultation with SAMHSA experts from CMHS and CSAT, and on pilot testing with the previous HIV Cohorts. Other SAMHSA HIV program experts, such as Jenifer Fiedelholtz of OPPB and David R Robertson of CSAT (among others), were consulted on the following issues:

• Draft study design plan and privacy/data security procedures; plan for coordinating and collecting data; measures to be used to assess outcomes; and mediating factors

• Suitability of proposed assessment instruments

• Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site study and reporting study findings

• Means of minimizing the burden on project staff and program participants

• Identification of efforts to ensure user relevance of results.

c. Consultations Outside of the Agency

CSAP consulted with other experts on SA, HIV/AIDS, and hepatitis as well as other Federal agencies with related programs or mandates, including NIDA, ACF, CDC, DOJ, OJJDP, HUD, and the DOE. Consultations resulted in the refinement of measures and the coordination of Federal data needs. For example, Kevin O’Conner, an expert on HIV prevention at the CDC, was consulted, and the following issues were discussed:

• Evaluation design and data privacy procedures

• The plan for coordinating and collecting data and measures to be used to assess outcomes and mediating factors, including:

  • Suitability of proposed assessment instruments

  • Materials and nuances of prevention programs that may be relevant to finalizing the methods to be used in conducting the cross-site evaluation and reporting study findings

  • Means of minimizing the burden on project staff and program participants

  • Efforts to ensure user relevance of results.

9. Payment to Respondents

No cash payment will be made to individual program participants from whom data will be collected. Although not a project requirement, some grantee organizations provide in-kind incentives to respondents (such as gift certificates from local vendors), for completing the evaluation study. The decision to provide incentives is left to the discretion of local sites.

10. Assurance of Confidentiality

CSAP has designed the multi-site questionnaire data collection strategy so that no identifying information such as names or complete Social Security Numbers will be requested of participants. Instead, each participant will be assigned a unique Individual Identifier number, as explained in the following paragraph.

To ensure privacy and security, grantees will assign each survey respondent a five-character code, or Individual Identifier (part D below). Individual Identifiers will begin with either a Y (assigned to those youth participants under 18 years) or an A (assigned to those adult participants 18 years and older) followed by a randomly generated four-digit number (ex.Y2942). The four values following the Y or A will be numeric values (not alphanumeric) and only used once per grantee. Numbers can range from 0001 to 9999. The unique ID number sequence is displayed as follows:

A B C D

[ ] [ ] [ ] – [ ] – [ ] – [ ] [ ] [ ] [ ] [ ]

For example, a survey respondent might have the ID number: 601-1-2­-A3543.

This number indicates that site 601 (A) recoded adult participant A3543 (D) as a member of the treatment group (B) and receiving a group intervention (C) at the time the form was completed. To further ensure the privacy and security of individual responses, all data will be reported at the aggregate level so that individual responses cannot be identified; no data will be reported at the individual participant level.

The data collection anticipated through this study falls under the Federal regulations regarding protection of human subjects in research (45 CFR 46). For all data collection concerning participating youth, informed written consent by parents and assent from the youth will be obtained. The evaluation will involve collecting data from local and re-entry minority youth and adults in communities residing in communities of color. They will complete a self-report questionnaire at three points in time, program entry, program exit, and three to six months post program exit. Several actions will be taken to protect their identities, including:

• All data collected will be maintained in a safe and private manner. The DACCC and grantees will conform to all requirements of the Privacy Act of 1974 under the System of Records, Alcohol, Drug Abuse and Mental Health Epidemiologic Data, HHS/SAMHSA/OA, #09-30-0036.

• Grantees will not send identifying information (i.e. name of respondent) to the DACCC. Only a questionnaire identification number will be provided. In addition, grantees will not provide identifying information to CSAP.

• Access to the data will be limited to the DACCC staff directly involved in the evaluation. At the end of the grant, a public use data diskette or CD-ROM will be made available containing the HIV program grantees’ findings, along with detailed documentation. These public use data files will contain no individual identifiers. Reports prepared by the DACCC as contract deliverables will present data in aggregate form only.

• All DITIC and DACCC staff will take a pledge agreeing that all information provided by respondents will be maintained with complete privacy and security.

11. Questions of a Sensitive Nature

Survey instruments include questions on ATOD use and attitudes, because these questions are necessary to obtain data that will help explain observed program outcomes. The proposed survey instruments incorporate all of the adult and youth NOMs items (OMB # 0930-0230) for GPRA and additional questions related to HIV/AIDS and hepatitis. Since HIV/AIDS is a sexually transmitted disease, these survey instruments include a number of questions of a sensitive nature, such as questions on sexual behavior and practice and their relationship to SA. These questions must be asked because the interventions are focused on use of safe sexual practices, the use of condoms, and the reduction of unsafe sexual practices due to the contributing influence of ATOD use on acquiring HIV/AIDS and Hepatitis C.

Grantees will routinely obtain informed consent from parents of youth participating in the study. Written, informed consent will be a necessary prerequisite at every grantee site prior to data collection. Grantees will guarantee that all data submitted to the contractor for the DACCC has first received the appropriate written consent. This consent will also indicate data collection and release to the DACCC. These consent forms, unique to each grantee, will specify both the risks and benefits of study participation. There are no data elements in the data collection instruments covered by the consent that will fall outside of this protection.

12. Estimates of the Annualized Hour Burden to Respondents

Table 1 shows the estimated annualized burden for data collection. The evaluation data will be collected through questionnaires administered to youth and adult program participants. Each youth and adult will complete questionnaires three times, taking an average of 50 minutes for baseline, exit, and follow-up questionnaires. Approximately 35,300 adults and youth are expected to respond at baseline. It is expected that 26,475 (75%) will respond to the exit questionnaire; and a total of 17,474 (66%) will respond to the follow-up questionnaire. Based on the HIV Cohort 3, 4, and 5 results, the expected response rates at exit are 74% and 61% at the three to six month follow-up. The total burden is 64,205 hours and the average annualized burden is 12,841 hours for the five-year program evaluation study, as noted in the following table.

Table 1. Estimated Annualized Burden of Data Collection

	Number of Respondents at Baseline	Number of Respondents at Exit	Number of Respondents at Follow-up	Average Burden/ Response (Hrs.)	Total Burden Hrs.	Hourly Wage Cost ($)	Total Wage Cost ($)
Total of Adults and Youth	35,300	26,122	15,934	0.83	64,205	­­­­­­­­$6.75	$433,387

The burden estimate presented in Table 1 is based on pilot test experience. There will be no direct cost to youth or adults for participating in the study. The value of youth and adult time was assumed given the prevailing minimum wage rate in California (California was chosen since it is often the “bellwether” for setting precedents later adopted by other States).

13. Estimate of Annualized Cost Burden to Respondents

There will be no capital, start up, or operation and maintenance costs incurred by the adults and youth participating in this study. The CSAMS on-line data collection tool will be available at no cost to grantees.

14. Estimate of Annualized Cost to the Federal Government

The total contract award for the DACCC to cover all aspects of the study design, planning, data collection, and analysis is $182,000 over a 42-month period. Thus, the annualized cost is $52,000. These costs cover the following activities: study design and planning; instrument development and pilot testing; assistance to study sites in cooperation with the national evaluation; processing of outcome data from study sites; data analysis and reporting; and development of public use data and documentation.

In addition, the annual cost for IT activities related to this data collection and data management is estimated at $180,000. These activities, provided by the DITIC, include:

• The provision of internet-based questionnaire data submission through CSAMS;

• Website and database maintenance, data backup, user account management, and Web security;

• System update, bug fixing, system enhancement, new module development, and system integration; and

• Training and technical support.

Additional costs will be incurred indirectly by the government in personnel costs of staff involved in oversight of the program evaluation study. It is estimated that 11 CSAP Project Officers will be involved for approximately 100 percent of their time at an average annual salary of $110,000. Direct costs of CSAP Federal staff time will approximate $1,210,000 annually. The annualized total cost to the government will be $52,000 + $180,000 + $1,210,000 = $1,494,000.

15. Changes in Burden

This is a new data collection.

16. Analysis Plans, Publication Plans, and Time Schedule for the Project

Analysis Plans

The defining characteristic of this cross-site study is the sharing of a common protocol, a common set of performance measures, common outcome objectives, and common evaluation questions by all participating grantees. This study differs from more traditional multi-site clinical trials because each individual grantee will select an EBP that is adapted to the needs of the particular target population, setting, and organizational characteristics. This multi-site evaluation does not test a single intervention that has different settings, rather it is testing a category of interventions that have similar outcome objectives but that use different approaches to accomplish those objectives.

Analysis of a multiple-site data set requires a complex set of interrelated tasks. Planning for these tasks must be flexible, and must allow adjustments as the opportunities and challenges presented by the empirical realities of the data set are discovered. While multi-site studies provide strong opportunities for knowledge generation (because of the ability to contrast intervention and implementation variation in a single study), they also present significant research challenges. This study recognizes those challenges and anticipates solutions as they will apply to the 81 participating grantees.

Sample Size Determination. Individual grantees have proposed their target population sizes. The establishment of sample size at the grantee level depends to some extent on financial constraints for program intervention services, staff allocation, participant retention activities and evaluation activities, including stipends. The overall sample size across all sites is shown in Table 2.

Table 2: Summary of Study Sample Size

Total Population	Total Population at Entry	Total Population at Exit	Total Population at 3-6 Month Follow-up
Total Adults & Youth	35,300	26,122	15,934

Statistical Procedure Determination. As a multi-site design, the SAMHSA/CSAP initiative collects information at two levels of observation: 1) across individuals, and 2) within individuals at three points in time. The units of observation also have a hierarchical relation. Individual level units (youth and adults) are nested within program sites and points in time are nested within individuals.

The proposed analysis includes several distinct steps:

• First, pooled analyses of outcomes will be conducted to assess the (controlled) presence of significant factors in growth curve trends for youth and adults participating in prevention interventions.

• Second, the heterogeneity of outcomes across sites will be assessed to determine if outcomes for substance use or important protective factors significantly differ across sites. If there are significant differences, hypotheses will be developed to explain those differences and conduct multivariate analyses on: a) clusters of sites that share characteristics hypothesized to be contributors to effectiveness, and/or b) individual sites that exhibit combinations of principles and practices hypothesized as contributors to effectiveness.

• Additional analyses will test the sensitivity of effectiveness models to differences in participant characteristics.

Statistical Test Determinations. Both the structural equation model (SEM) approach to estimating the trajectory parameters and the hierarchical linear model (HLM) approach that can consider time to be nested within an individual will be the key analytic methods conducted for this multi-site evaluation (MSE) (Bollen, 1989; Chou, Bender, & Pentz, 1998). This hierarchical data set presents flexible analysis opportunities as well as some analytic challenges. As the primary statistical tool, the plan is to apply multi-level regression [e.g., SEM and HLM] models. This technique allows for the identification of individual effects, controls for co-variates (e.g., propensity scores to control for non-equivalence across intervention groups), and tests for interaction effects with the different types of interventions and youth or adult characteristics that may mediate the impact of the intervention. HLM also provides excellent capacity for analyzing longitudinal, repeated measures designs (Willett, Singer, and Martin, 1998), can accommodate missing data at individual data points, and allow adjustments for different individual intervals between follow-up data points. Tests for attrition bias and selective attrition are conducted at each follow-up point.

A two-step analysis strategy will be used to control for attrition bias. First, baseline characteristics of all participants will be compared with baseline characteristics of participants who completed the programs. These two sets of data records will be compared with respect to baseline values of demographic characteristics, incarceration/reentry status, levels of risk and protection, and levels of substance use. Second, baseline factors found to differ significantly between the two sets of records will be included in all of the models and only model estimates net of these factors will be reported in the final evaluation results. This approach minimizes selectivity biases due to program attrition in reported program effects.

While only some of the 81 program sites are incorporating a more robust experimental design with control/comparison groups, the MSE dataset will provide the flexibility to conduct analyses that provide useful evidence concerning the general effectiveness of prevention in reducing risk factors and promoting (developing) protective factors that could potentially affect SA or HIV/AIDS or hepatitis risk or transmission. The dataset will also be able to assess the variation in this effectiveness between interventions among the project sites, and offer potential explanations of that variation (e.g., amount of contact, type of intervention approach).

Given that target populations and community contexts vary by grant site, the multi-site dataset will contain data from a wide range of program participants in terms of their demographic, socioeconomic, and cultural characteristics. All multivariate models will include all of the factors that account for the differences among groups. Those factors that are found to have a significant effect on outcomes will be identified and interaction terms will be constructed to represent differences in program effects due to recruitment strategy. This analytic strategy will allow the evaluation study to take into consideration the mediating effects of a broad range of factors on program outcomes. The inclusion of these demographic, socioeconomic, and cultural control variables and interaction terms in the models will also ensure that final results are not biased toward the outcomes of groups with relatively large numbers of data records.

Many of the grantees are targeting African American and Hispanic/Latino populations and populations who have just been released (re-entry) from the criminal justice system. Youth between the ages of 12 and 17 and adults aged 18 and over are included in the cross-site study design. Varieties of methods are being used to recruit participants. Most sites report that participants will be identified from SA treatment programs, referrals from collaborations with criminal justice systems and other involved agencies, and/or through site-specific geographical areas that were identified through the SPF Step 1 needs assessment conducted during the first year of the grant. This needs assessment will identify key risk factors, including risk for substance use, HIV/AIDS or hepatitis transmission, and economic disadvantage. Step 2 will assess capacity to provide services. For Steps 3 and 4 (planning and implementation), grantees are allowed to select and adopt a variety of evidence-based prevention intervention approaches to fit the needs of their program participants/clients.

A multi-level analysis approach [e.g., SEM and HLM] will be used to investigate the effects of program characteristics on participant outcomes. Characteristics hypothesized to have a bearing on program effects, such as choice of prevention strategy and type of grantee organization, will be included in the dataset together with participant-level baseline, exit, and followup survey data. Nesting participant-level data within program-level data in this fashion will allow the construction of multi-level causal models that simultaneously test for the effects of participant and program characteristics on program outcomes and to identify significant interactions between these two levels.

Publication Plans

The MAI cross-site study results will be made available to the public through publications and conference presentations. The following journals carry articles on SA prevention and HIV/AIDS and are expected to serve as potential vehicles for distribution of study results: Journal of Substance Abuse Treatment, International Journal of Addictions, Journal of Community Psychology, Journal of Adolescent Research, Journal of Adolescent Health, Preventive Medicine, Evaluation Review, Policy Studies Review, and the American Journal of Public Health. Study results could also be published in other journals that focus on HIV/AIDS. These include The Journal of the American Sexually Transmitted Disease Association, Health Education and Behavior, AIDS: Official Journal of the International AIDS Association, AIDS Education and Prevention, The Journal of Sex Research, AIDS Care, Psychological and Socio-Medical Aspects of AIDS/HIV, and Current Opinion in HIV and AIDS. Study results also are targeted for publication in journals focusing on infectious diseases. These include, among others, The Journal of the American Microbiological Association and Journal of Infectious Diseases.

The study results will be distributed through presentations at annual conferences of national and international public health organizations, such as the Society for Prevention Research, the American Public Health Association, the National Association of Alcohol and Drug Abuse Counselors, The National Prevention Network, the American Evaluation Association, and HIV/AIDS national meetings as well as regional and State SA prevention and treatment associations. HIV/AIDS meetings could include, among others, CDC Annual Conferences on AIDS and Conferences of the International AIDS Society. Results could also be presented at meetings focusing on infectious diseases such as annual meetings of the American Society of Microbiology.

Documents will also be prepared and published on behalf of the government (CSAP/SAMHSA) through the Government Printing Office (GPO) for Federal agency and public use. Findings will also be available via OMB’s Website: www.expectmore.gov, as well as in annual reports to Congress and the performance detail sections of annual SAMHSA budgets as they become publicly available.

Timeline

The MAI is a five-year grant program (see Table 3). Years I and 2 are devoted to Steps 1, 2 and 3 of the SPF, namely conducting the needs assessment, capacity building, and planning, respectively. Years 3, 4, and 5 are devoted to Steps 4 and 5, implementation and evaluation, respectively.

Table 3. Project Timeline

Date	Activity
FY2007	Needs Assessment, Capacity Building, Planning
FY2008	OMB clearance obtained, Implementation
FY2009	Implementation
FY2010	Analysis and Reporting

17. Display of Expiration Date

The expiration date will be displayed.

18. Exceptions to Certification Statement

No exceptions are required.

REFERENCES

Bentler, P.M., (1995). EQS Structural Equations Program Manual. Encino: Multivariate Software, Inc.

Bollen, K.A. (1989). Structural equations with latent variables. New York: Wiley.

Chou, C.P., Bentler, P.M., Pentz, M.A. (1998). Comparison of two statistical approaches to the study of growth curves: The multilevel model and latent curve analysis. Structural Equation Modeling, 5, 247-266.

Koziel, M.J., and Peters, M.C. (2007). Viral hepatitis in HIV Infection, M.D., NEJM 14: 356:1445-1454.

Springer, F. (2000). "Lessons learned from multiple cross-site evaluation studies.” Paper presented at the American Evaluation Association Meeting, Honolulu, November 2000.

Straw, R.B. & Herrell, J. M. (2002). Conducting multiple site evaluations in real-world settings. San Francisco, CA: Jossey-Bass.

Tedaldi, E.M. et al., (2003). Prevalence and characteristics of Hepatitis C coinfection in Human Immunodeficiency Virus Clinical Trails Group: The Terry Beirn Community Programs for Clinical Research on AIDS, Clinical Infectious Diseases, 15:36(10):1313-7

Willett, J.B., Singer, J.D., & Martin, N.C. (1998). The design and analysis of longitudinal studies of development and psychopathology in context: Statistical models and methodological recommendations. Development and Psychopathology, 10, 395-426.

Winstock, A.R., Anderson, C.M., and Sheridan, J. (2006). National Survey on HIV and Hepatitis Testing and Vaccination Services provided by Alcohol Agencies.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The plan is to sample the whole universe and based on this sample, have grantees propose their target population sizes. The establishment of the sample size at the grantee level depends to some extent on the size of their client population, the financial constraints on the program intervention/treatment services, staff allocation and retention, and the availability of stipends for data collection activities. Table 2 displays the aggregate baseline sample sizes for program participant groups. As noted, the expected response rates at exit are 74% and at 61% at the 3-6 month follow-up, based on the HIV Cohort 3, 4 and 5 results.

2. Information Collection Procedures

Common Measures - Youth and Adult Questionnaires (Completed by Program Clients)

The study uses a common protocol for collecting program and participant/client level data and submitting it to SAMHSA via an online, web-based data entry system (CSAMS). The DACCC Technical Assistance Center also has been established to assist grantees with online data entry. These data provide the basis for categorizing programs on characteristics of program design and implementation (e.g., one-on-one, group format). These are described below:

Youth and Adult Outcome Questionnaires. Two common questionnaires will be administered to program participants/clients. Each questionnaire has three versions, as follows:

• Youth Questionnaire: For persons aged between 12 and 17 - Baseline, Exit, and Follow-up

• Adult Questionnaire: For persons aged 18 and older - Baseline, Exit, and Follow-up

For all common measures, administration guides have been prepared to assist program sites with implementation.

The major constructs for the youth outcome questionnaire include demographics, 30-day substance use (ATOD), age of first use, disapproval of ATOD use, perception of risk, ATOD use, family cohesion, perception of peer behavior, sexual behavior, school connectedness, knowledge of HIV and hepatitis, and health care (See Attachment 1 for a copy of the Youth Baseline Questionnaire and the Administration Guide.)

The major constructs for the adult questionnaire include demographics, employment, 30-day substance use (ATOD), age of first use, disapproval of ATOD use, perception of risk, ATOD use, age of first use, family cohesion, sexual behavior, knowledge of HIV and hepatitis, health care. Questions for both the adult and youth questionnaires do not change across the three data collection points (e.g., baseline, exit, and follow-up) (See Attachment 2 for a copy of the Adult Baseline Questionnaire and the Administration Guide.)

While the GPRA and NOMs measures have already been approved by OMB (OMB No. 0930-0230), the remaining HIV-related questions have not, hence this application for OMB Clearance. The questionnaires contain 135 questions, of which 102 relate to HIV/AIDS and hepatitis. The planned analysis will assess the degree to which program strategies reduce SA/HIV/AIDS and hepatitis risk and increase SA, HIV/AIDS, and hepatitis protective factors among minority and minority re-entry populations.

The information obtained from these questionnaires will generate data to determine the effectiveness of the program in reducing SA, injection drug use, and high-risk sexual behaviors as well as increasing participant knowledge. Pilot tests have been conducted to assist in controlling respondent burden and ensuring the user-relevance of questions.

Data Collection

Time Points (Efficacy) for Data Administration: The common design includes assessments at baseline, program exit, and three to six months post-exit (follow-up). The common questionnaires will be administered to all intervention (program participants) youth and adults at baseline (first data collection point), program exit (second data collection point), and follow-up (third data collection point).

• Baseline Questionnaires: This questionnaire (first data collection point), should be administered within 30 days of intake or before core program services begin.

• Exit Questionnaires: This questionnaire (second data collection point), should be administered again within 10 days post program exit or after core program services have ceased.

• Follow-up Questionnaires: This questionnaire (third data collection point), should be administered within 30 days of the planned follow-up (post-exit). Most study sites have a planned follow-up administration for three to six months after program exit.

The common instrument assessment schedule is depicted in Table 4.

Table 4. Common Instrumentation Assessment Schedule

Instrument	Baseline	Exit	Follow-up
Youth Questionnaire	X	X	X
Adult Questionnaire	X	X	X

This time series analysis is depicted graphically below:

01 X 02 03

Where: 01= Baseline - First Data Collection Point

X = Program Prevention Intervention Activities

02 = Exit - Second Data Collection Point

03 = Follow-Up – Third Data Collection Point

Sample size, respondent burden, and intrusiveness have been minimized to be consistent with cross-site analytical objectives. Procedures are employed to safeguard the privacy and security of participant responses. Every effort has been made to coordinate cross-site data collection with local data collection efforts to minimize respondent burden. Pilot study results indicate that the questionnaires require an administration time of approximately 50 minutes.

Data Management

Storage System: Over the life of this initiative, each grantee will be collecting information that must be documented and organized. Each local evaluation team or the person responsible for data management will be required to store:

• Individual questionnaires (Youth and Adult and Individual and Group Dosage) until they are entered or uploaded into CSAMS

• Consent forms

• Tracking forms for each study participant.

Before initiating data collection, each site is responsible for setting up a filing and storage system that will accommodate these needs.

Web-Based Data Entry Upload System: The DACCC has created CSAP’s Services Accountability and Monitoring System (CSAMS), an online data entry system which provides prevention information, data collection tools, documents, data entry functions, and access to reporting statistics and tracking. All of the HIV instruments can be found in the “Tools” section of this website. Common questionnaires are available in both Microsoft Word and PDF format for individual grant sites to download and make copies for administration to clients or participants. Site evaluators or data collectors are expected to enter client or participant responses to questionnaires through the CSAMS website. Sites will also be able to upload response databases through CSAMS that use the appropriate variable/value numbering (questionnaire codebooks are also available on the “Tools” section of CSAMS website). Dosage data will also be entered online via the CSAMS website by the project director or evaluator of the grant site.

Once data have been entered into CSAMS, the DACCC will clean the data and each grant site will have access to their downloadable, clean, electronic data files. Once data are entered by the grant site and cleaned by the DACCC, data will be available for download by the grant site for use in local data analysis and reporting. Sites will be able to enter the data online, or upload if necessary, on a continual basis. The DACCC will abstract data bi­annually for cleaning, analysis, and reporting purposes; however the data will remain accessible for local evaluations.

CSAP has established a technical assistance center to assist program sites. Any questions regarding the common instruments or submission to the DACCC should be addressed to the Technical Assistance Hotline for CSAMS, available Monday through Friday, from 9 a.m. to 8 p.m. Eastern Standard Time via telephone, (240) 223-3002 or (877) 654-6740, or via email, [email protected].

3. Methods to Maximize Response Rates

Based on the HIV Cohort 3, 4, and 5 results, the expected response rate at exit is 74% and 61% at three to six month follow-up. As noted above, each grantee has developed a package of incentives to maximize their response rates. These incentives have been selected on the basis of each grantee’s past experience with their client population. Typically, prevention programs have lower retention rates than treatment programs, so these estimates fall below the OMB expected norm of 80%. In order to achieve these retention rates, it is also necessary to provide incentives.

CSAP's program officers will monitor monthly participation numbers at baseline and exit through the online CSAMs system. The CSAMs system is the online data collection tool that grantees use to upload their qualitative and quantitative data. This system will also be used as a communication tool for grantees to identify to the program officer any potential barriers to retention of participants. Furthermore, CSAP provide annual training and at hoc training to programs dealing with the retention of participants.

4. Tests of Procedures

CSAP and its contractor have reviewed the adult and youth questionnaires, clarifying terminology and language, and rewriting or eliminating unclear or unnecessary questions. Questionnaires were then pilot-tested on a small sample (less than 10 individuals) to ensure that the multi-site assessment requirements and procedures were consistent with activities conducted at education sites. Pilot-testing was designed to collect information on the overall evaluation design and to draft feedback forms.

All study sites were encouraged to participate in pilot testing because of the diversity in target populations and program settings. (Note: It was only after local sites had completed their pilot testing that project staff learned that not more than nine persons could be included in testing without OMB approval). Pilot testing included administration of draft instruments and solicitation of comments on the instrument from respondents.

The purpose of the pilot testing was to:

• Identify instructions and/or questions within the instrument that were unclear or confusing

• Obtain suggestions for improving questions or instructions

• Identify data collection procedures that were unclear or confusing

• Measure respondent burden based on the average time to complete instruments

• Identify ways to minimize respondent burden and improve accuracy regarding instrument completion.

Comments on the draft feedback forms included collecting information on the likelihood of obtaining specific responses, overall instrument layout, item flow, and administration.

5. Statistical Consultants

Virginia Mulkern, Ph.D.

Senior Vice President

Human Services Research Institute

2336 Massachusetts Avenue

Cambridge, MA 02140

Nilufer Isvan, Ph.D.

Senior Research Fellow

Human Services Research Institute

2336 Massachusetts Avenue

Cambridge, MA 02140

P. Allison Minugh, Ph.D.
President/CEO
Datacorp, Inc.
Two Richmond Square, Suite 100A
Providence, Rhode Island 02906

ATTACHMENTS

1. Youth Questionnaire and Youth Questionnaire Administration Guide

2. Adult Questionnaire and Adult Questionnaire Administration Guide

Minority HIV/AIDS Initiative Page 25