Access to Recovery (ATR) Program Cross-Site Evaluation
Office of Management and Budget
Supporting Statement and Attachments
Submitted by:
RTI International
TABLE OF CONTENTS
Section Page
A.1 Circumstances of Information Collection 1
A.2 Purpose and Use of Information 2
A.2.1 Client Survey (Attachment 3) 3
A.2.2 Provider Survey (Attachment 4) 4
A.3 Use of Information Technology 5
A.4 Effort to Identify Duplication 6
A.5 Involvement of Small Entities 6
A.6 Consequences If Information Collected Less Frequently 6
A.7 Consistency with the Guidelines in 5 CFR 1320.5(d) (2) 7
A.8 Consultation Outside the Agency 7
A.10 Assurance of Confidentiality 9
A.11 Questions of a Sensitive Nature 11
A.12 Estimates of Annualized Hour Burden 12
A.13 Estimates of Annualized Cost Burden to Respondents 13
A.14 Estimates of Annualized Cost to Government 13
A.16 Time Schedule, Publications, and Analysis Plan 14
A.17 Display of Expiration Date 16
A.18 Exceptions to Certification for Statement 16
B.1 Respondent Universe and Sampling Methods 17
B.2 Information Collection Procedures 18
B.2.2 Client Survey Information Collection Procedures 20
B.2.3 Provider Survey Information Collection Procedures 22
B.2.4 Provider Survey Mail Data Collection Procedures 23
B.3 Methods to Maximize Response Rates 24
Exhibits
Number Page
Exhibit 1. Expert Panel Members 7
Exhibit 2. Cross-Site Data Collection Burden for Client Survey 12
Exhibit 3. Estimates of Annualized Cost to Government 13
Exhibit 4. Time Schedule for Entire Project 14
Exhibit 5. Grantee Client Flow 18
Exhibit 6. Expected Respondent Sample by Grantee and ATR domain 19
Exhibit 7. Key Cross-Site Evaluation Team Members and Senior Advisors 26
ACCESS TO RECOVERY (ATR) PROGRAM CROSS-SITE EVALUATION
SUPPORTING STATEMENT
The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Center for Substance Abuse Treatment (CSAT) requests approval from the Office of Management and Budget (OMB) for the data collection activities of the “Access to Recovery” (ATR) Program Cross-Site Evaluation. Data collection consists of two surveys: the ATR Client Survey and the ATR Provider Survey. The client survey will be administered to individuals who have received or are receiving substance abuse services funded through the ATR Program. The provider survey will be administered to substance abuse service provider organizations participating in the ATR Program.
The ATR Program is authorized under The Public Health Service Act (sections 501(d) and 509; 42 U.S.C. sections 290aa [d] [5] and 290bb-2). This program also addresses Healthy People 2010, Volume II (Part B: Focus Area 26 Substance Abuse). As a Presidential initiative, the ATR Program is the only Federal program to require client choice among a network of substance abuse service providers, including faith-based and community organizations. As such, the program contributes to meeting the goals of the President’s Faith-Based and Community Initiative.
Most substance abuse treatment funding programs, including those administered by SAMHSA/CSAT, have not directly facilitated multiple pathways to recovery that include access to faith-based and community providers and recovery support services [RSS] or genuine consumer choice among providers. Initial research suggests that facilitating multiple recovery options and choice may positively impact an individual’s decision to enter treatment, their ability to obtain services, and the overall outcomes. Offering a wider choice of treatment options may improve the likelihood that clients will find a program matching their needs, thereby encouraging more clients to enter treatment and increase retention rates.
Recognizing that the needs for broader access and client choice could be met through a consumer-driven voucher program, SAMHSA/CSAT, in conjunction with the President’s initiative, established the ATR Program. The ATR Program is a competitive, discretionary grant program awarded to 18 States, the District of Columbia, and 5 Tribal Organizations to facilitate individual choice and promote multiple pathways to recovery through the development and implementation of a voucher-based financing system for clinical treatment and recovery support services (RSS).1
The ATR Program represents a significant change in Federal substance abuse treatment financing. With vouchers, the ATR Program allows clients to obtain RSS and to access faith-based and community organizations. This new funding mechanism for RSS and faith-based and community organizations may significantly affect client outcomes, access to services and subsequently expand substance abuse treatment capacity. Because of the uniqueness of the ATR Program, it is critical to fully understand how the program performs and which aspects of the program offer the greatest potential to reshape the substance abuse treatment system.
Currently, SAMHSA/CSAT monitors the performance of the ATR Program using data collected through the Government Performance and Results Act (GPRA) (OMB No. 0930-0208). Although GPRA data are sufficient for program monitoring, they are not sufficient for understanding client satisfaction and choice. Furthermore, the provider survey provides SAMHSA/CSAT with specific details on ATR providers’ organizational characteristics and experiences and satisfaction with the ATR program. The client and provider surveys will provide data necessary for a comprehensive evaluation of the ATR Program on both the client and provider levels.
The purpose of the data collection is to conduct a cross-site evaluation of the ATR Program. The ATR client and provider surveys directly help SAMHSA/CSAT achieve a number of specific internal and external goals for the ATR Program. For CSAT, client choice is a key expectation of all ATR grantee programs. As outlined in the ATR Request for Applications, Section 2, Expectations, grantees are mandated to provide every ATR client a choice from at least two providers including at least one provider to which the client does not object for religious reasons (Attachment 1). The client survey, through its questions on client choice and experience, will allow CSAT to better understand the degree of choice offered from the client’s perspective. Similarly, CSAT expects grantees to develop and operate a voucher system that ensures client satisfaction (see Attachment 1). The client survey provides CSAT with systematic and standard data across all grantees that will allow the cross-site evaluation to evaluate client satisfaction and its potential mediators and moderators (e.g., client characteristics, client outcomes, grantee characteristics).
Section 2 of the ATR Request for Applications also lists participation of faith-based and community organizations and expansion of clinical treatment and RSS as expectations of the grantees (see Attachment 1). The data collected with the provider survey will allow CSAT to more fully evaluate these expectations from the provider’s perspective. This information will contribute to determining possible moderators and mediators of provider organizations including faith-based and community organizations and the expansion of clinical treatment and RSS within the treatment environment targeted by the ATR Program.
The provider and client surveys directly contribute to fulfilling a need identified by the Program Assessment Rating Tool (PART) for an independent evaluation of sufficient scope and quality. The 2007 PART improvement plan for the ATR Program included a proposed independent cross-site evaluation to provide the needed information (Attachment 2). Some of the additional data requested by PART will come directly from provider and client survey data.
In addition to these specific goals, the client and provider surveys will contribute data necessary for a comprehensive evaluation. The following paragraphs describe the surveys.
Although GPRA data are collected from ATR clients, the GPRA instruments do not collect data on client choice, experience, or satisfaction. Thus, the client survey captures data not already obtained through CSAT’s monitoring efforts. Moreover, the client survey data will be linked to GPRA data allowing for client characteristics and outcomes to be included, when possible, in the analysis of client survey data. This will provide CSAT with a more comprehensive picture of grantee, provider, and client-level factors that may influence client choice, experience, and satisfaction.
The targeted universe for the ATR client survey are individuals who have or are receiving ATR-funded services and are eligible for a 6-month post-intake GPRA follow-up across the 24 funded grantee programs. Client eligibility for ATR funding varies by grantee program as each grantee designed its respective ATR Program to best address local needs. The client survey will be administered once to a sample of ATR clients approximately 5 to 8 months post-intake to coincide with the 6-month post-intake GPRA follow-up. The client survey instrument is a paper-and-pencil, self-report survey.
The client survey contains items from the Mental Health Statistics Improvement Program (MHSIP-28) Consumer Survey to measure client satisfaction. In addition, questions on client choice and experience have been developed based on a review of grantee client surveys; various established data instruments drawn form the literature, and expertise of the cross-site evaluation team.
Substance abuse service provider organizations (e.g., residential substance abuse treatment facilities, organizations hosting Alcoholics Anonymous meetings, individual substance abuse treatment practitioners) participating in a grantee’s ATR provider network are eligible for the ATR provider survey. The provider survey collects data on organization characteristics, satisfaction, and experience with the ATR Program. The provider organization will be asked to complete the survey via a Web instrument. At each provider organization, the survey targets a key informant, typically a director or manager within the organization most knowledgeable on the ATR Program. The sampling frame is a census of all providers participating in a grantee’s ATR network as of December 31, 2008. Any providers joining a grantee’s ATR provider network after this date will not be eligible for the survey.
The provider survey was developed to specifically address the unique provider organization experiences associated with participation in a grantee’s ATR network. Survey questions are based on the cross-site evaluation’s provider-level evaluation questions and developed by experts on the ATR cross-site evaluation team.
The client survey will be administered as a paper-and-pencil, self-report instrument. This method is used to reduce the burden on ATR grantees and ensure respondent privacy. The survey form will use electronically scannable TeleForm technology to reduce data entry burden and errors. Alternative methods of administering the client survey that use specialized information technology were considered. The following outlines the technologies considered and the reasons they were not adopted:
Web survey: The target population and the survey’s administration time frame make this method exceedingly difficult and subject to significant bias. The target population is unlikely to have reliable access to the Internet across all potential respondents. Moreover, because the survey is administered 6-months post-intake, it is impractical for a substance abuse service provider to facilitate client access to a Web survey. Any responses obtained via this method would be unlikely to represent a typical client and suffer from low response rates.
Computer-assisted telephone interview (CATI): Under special circumstances, GPRA follow-ups may be administered via the telephone. However, the client survey will not be administered via CATI to ensure respondent privacy, consistency in administration, and unbiased data.
Computer-assisted personal interview (CAPI): Grantee representatives participating on the Expert Panel2 clearly stated that accessing clients via CAPI would be exceedingly difficult and cost prohibitive. The Panel recommended that survey distribution be coordinated with the collection of the 6-month post-intake GPRA follow-up to reduce respondent burden and overall cost. This method requires a paper-and-pencil form.
Given the input provided by the Expert Panel, the need to ensure client privacy, and the realities of accessing the targeted population, the client survey will require a paper-and-pencil, self-report mode of administration. Special considerations have been taken to ensure that use of this method does not increase respondent burden. Specifically, the survey has a limited number of questions and no complex skip patterns.
The provider survey primarily will be administered through a Web instrument. See Attachment 5 for sample screens of the Web instrument. Using a Web instrument allows for automated skip procedures and automated fill-ins based on prior responses to certain questions, which will significantly reduce the burden among subsets of respondents. This method also uses automated data entry and greatly reduces the possibility of data entry error. Providers unable to complete the Web instrument will receive a paper version through the mail, and data entry will utilize specialized technology as appropriate (e.g., TeleForm). The provider survey Web instrument will comply with the requirements of Section 508 of the Rehabilitation Act.
The ATR cross-site evaluation team conducted an extensive literature review to confirm that the data collected through these surveys would not duplicate any ongoing national data collection efforts. Data collected by the client and provider surveys will be unique because of the specific and national focus on the ATR Program.
Linking to ATR GPRA data, allows the cross-site evaluation team to significantly reduce duplication of needed data elements. Basic respondent data such as race/ethnicity will be collected on the GPRA form and not on the client survey. This allows the client survey to focus on the domains of interest (client choice, experience, and satisfaction) not addressed in the GPRA interviews.
Individual grantees are 18 States, the District of Columbia, and 5 Tribal Organizations. Each of the grantees will utilize some small entity providers to provide treatment/services to clients, and some of these small entity providers may be involved in data collection. The short client survey form is specifically designed to limit any additional burden, and therefore no small entities will be significantly affected.
The provider survey will target some small entity providers, and special consideration has been taken to significantly reduce the burden on any small entity respondents. Using the Web instrument’s automated skip capabilities, many small entity respondents will be automatically asked to respond to fewer questions and therefore no small entities will be significantly affected.
Both the client and provider surveys ask respondents to respond only once. The information collected by the surveys is necessary for CSAT to monitor the achievement of multiple ATR-specific goals and for a comprehensive ATR cross-site evaluation (see Section A.2 for details). Without these data, CSAT will not be able to assess specified outcomes of the ATR Program and therefore would not be able to fully address concerns from the 2007 PART review.
This information collection fully complies with 5 CFR 1320.5(d) (2).
The notice required by 5 CFR1320.8 (d) was published in the Federal Register on July 22, 2008 (Vol. 73, No. 141, page 42584). No comments were received in response to this notice.
SAMHSA has made extensive use of experts in the area of substance abuse research to provide guidance on the design of the cross-site evaluation. The Expert Panel meeting was held in February 2008 to review the various aspects of the cross-site evaluation, including the evaluation plan, data collection procedures, the client and provider surveys, and literature review. The list of experts is provided in Exhibit 1. The experts provided feedback on all aspects of the evaluation, including the surveys, and their comments were incorporated into the survey design and administration. As noted, four of the panel participants represented grantee programs.
Exhibit 1. Expert Panel Members
Expert |
Affiliation |
Contact Information |
John W. Gastorf, PhD*
|
Cherokee Nation Behavioral Health Services |
1140 Mayberry Tahlequah, OK 74464 Phone: (918) 207–3898 Fax: (918) 458–7601 E-mail: [email protected] |
June Gertig, JD
|
Altarum Institute
|
1200 18th Street, NW, Suite 700 Washington, DC 20036 Phone: (202) 828–5100 E-mail: [email protected] |
William Halsey, MBA*
|
Connecticut Department of Mental Health and Addiction Services (DMHAS)
|
410 Capitol Avenue P.O. Box 341431 Hartford, CT 06134 Phone: (860) 418–6747 E-mail: [email protected] |
|
|
Cont. |
Cont. |
|
|
Expert |
Affiliation |
Contact Information |
Antoinette Krupski, PhD
|
CHAMMP Department of Psychiatry and Behavioral Sciences University of Washington Harborview Medical Center |
325 9th Avenue 2HH–1A Box 359911 Seattle, WA 98104 Phone: (206) 341–4215 E-mail: [email protected] |
Craig Love, PhD*
|
Westat
|
1650 Research
Boulevard Phone: (240) 314–2443 E-mail: [email protected] |
William E. Schlenger, PhD
|
Abt Associates, Inc. |
4620 Creekstone Drive, Suite 190 Durham, NC 27703 Phone: (919) 294–7712 Fax: (617) 386–8559 E-mail: [email protected] |
Laura Schmidt, PhD
|
Institute for Health Policy Studies and Department of Medical Anthropology, History, and Social Medicine University of California, San Francisco
|
Box 0936 3333 California Street, Suite 265 San Francisco, CA 94118 Phone: (415) 476–0440 E-mail: [email protected] |
Bert Singleton*
|
Peer Assistance Services
|
2170 South Parker Road, Suite 229 Denver, CO 80231 Phone: (303) 369–0039 ext. 203 E-mail: [email protected] |
Chris Tompkins, PhD
|
Brandeis University Heller School for Social Policy and Management
|
415 South Street, MS–035 Waltham, MA 02454–9110 Phone: (781) 736–3913 Fax: (781) 736–3905 E-mail: [email protected] |
*Representative of an ATR grantee program
Clients will be provided an incentive (e.g., $10 gift card) upon completing the survey. An incentive will be given with the ATR client survey because the target population is a hard to reach population and an incentive is expected to significantly increase response rates. The incentive size balances the expected impact on responses rates with the estimated low burden time (estimated at 6 to 10 minutes). Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. Substance abuse research has shown improved response rates when remuneration is offered to respondents. Individuals with substance use issues are typically harder to reach, and the potential payment at follow-up should lead to a higher follow-up rate.
Results from the 2001 National Household Survey on Drug Abuse (NHSDA) incentive experiment were reported by Wright et al. (2002). They found that the $20 and $40 incentive payments each produced about a 10-point gain in overall response rates when compared with the $0 control group. The overall response rate was significantly higher for the $40 incentive than for the $20 incentive within many of the subgroups addressed in the analysis. Both incentive payment groups more than paid for themselves due to decreased costs of follow-up and more productive screening resulting from the improved response rates.
Respondents will not receive payment.
Concern for respondent privacy and protection of respondents’ rights will play a central part in the implementation of all study components. RTI International (the contractor) has been contracted to administer the client survey with assistance from the ATR grantee programs in distributing the survey forms. Administering the survey includes receiving and analyzing raw, de-identified data from the client survey. The contractor is also administering the provider survey and analyzing the resulting data. The contractor has extensive experience protecting and maintaining the privacy of respondent data.
Additionally, the client and provider surveys and the administration protocols will be reviewed and approved by the contractor’s Institutional Review Board (IRB) (Federal-Wide Assurance Number 3331) and by grantee IRBs where applicable and necessary prior to survey implementation. In keeping with 45 CFR 46, Protection of Human Subjects, the survey procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent.
That all data will be private is explained to all respondents during the consent process and in the consent forms. The provider and client surveys’ informed consent procedures and documents clearly state:
that the survey is sponsored by an agency of the Federal Government,
the purpose and use of the information collection,
that providing the information is completely voluntary, with no penalties for no response, and
that respondent privacy will be maintained.
(See Section B.2, pages 20–24, for survey procedures. Attachments 6 and 7 also illustrate survey procedures as explained to participants.)
To help ensure respondent privacy the client survey does not collect any personal identifiers. A client’s GPRA identification number (ID) is recorded on the survey prior to distribution to link the client survey data with the ATR GPRA data. The GPRA ID is not a client birth date or Social Security Number; it is a unique ID number that does not compromise a respondent’s privacy. Additionally, the cross-site evaluation team will have no crosswalk data between GPRA IDs and personal identifiers.
The ATR grantee programs will not have access to survey responses as respondents complete the survey independently and seal responses before returning the completed surveys. ATR grantee programs distributing the survey are expected to follow the same standards of protecting and maintaining respondent privacy as applied to other data collections (e.g., GPRA OMB No. 0930-0280).
To help ensure respondent privacy, the provider survey does not directly link personal identifiers to responses and the survey does not ask for any personal identifiers for the individual completing the survey. The use of provider contact information and survey responses is described in the following paragraphs.
ATR grantees will provide publicly available provider contact information, which will include an organization’s address; telephone number; ATR-specific information, such as types of services provided; and name and e-mail address of a key informant at each provider. The provider organization’s contact lists will be maintained separately from the provider survey data. A randomly generated ID number will be assigned to each provider that will link provider contact information with the provider survey data.
Unique, secure user accounts will be created for each provider to access the Web survey. A provider organization’s user name will be the randomly generated ID number linked to the provider’s contact information. A secure password will be e-mailed (or mailed if the provider organization lacks a secure e-mail address) separately from the user name. A crosswalk dataset will be created from the provider contact list and will only include the provider’s random ID number, zip code, city, and state/grantee. This crosswalk dataset will be linked via the random ID number to the survey data to form the final provider survey dataset. Only select project staff from the ATR cross-site evaluation team will have access to the crosswalk database, the provider contact information and the final provider survey dataset.
All contact lists and all provider survey data will be stored separately on secure servers maintained by the contractor . Details on the contractor’s network security procedures are presented in Attachment 8. Once data collection is complete, all contact information will be purged.
The ATR client survey will not collect information that is personally sensitive. The survey asks questions regarding client choice, experience, and satisfaction with a grantee’s ATR Program. The survey will not ask clients to share information about substance use, mental health, or other health or social risks.
Informed consent will be obtain for participation in the survey, and the data collection procedures are developed to ensure that participants are fully informed and aware of their rights (see Section B.2, pages 20–22, for detailed survey administration procedures).
The client survey data will also be collected from minors. Parental permission will not be obtained because the survey only asks for only de-identified, nonsensitive data and obtaining parental permission would not be practical and potentially detrimental to participating minors. Minors will assent to participate following the same procedures used for adult participants. These procedures and the waiver of parental permission will be reviewed and approved by the contractor’s IRB (see Attachment 9).
No personally sensitive or proprietary information will be collected from providers, and the survey administration procedures ensure that respondents are fully informed and aware of their rights (see Section B.2 for detailed survey administration procedures, pages 22–24). Providers will respond to questions on organizational characteristics, an organization’s experience as a part of a grantee’s ATR Program, and overall satisfaction. The survey will not collect information that would negatively impact the organization or the individual completing it. As outlined in Section A.10, contact information and responses will be maintained to protect respondent privacy.
Estimate the annualized hour burden of the collection of information from clients. As data collection will be completed within 1 year, the following estimates represent the total time burden for the survey. The total client sample size for the cross-site data collection effort is estimated to be 7,329 respondents aged 12 and older. Exhibit 2 presents estimates of total burden based on preliminary testing. Although average client burden is used, the time required to complete the surveys will vary slightly depending on client characteristics. Average time required to hear the survey script, receive supporting forms, and complete the survey is estimated at 9 minutes with a range of 6 to 11 minutes.
Estimate the annualized hour burden of the collection of information from providers. As data collection will be completed within 1 year, the following estimates represent the total time burden for the survey. The total number of providers eligible for the survey is estimated at 5,104, of which 4,083 (80%) are expected to respond. Average burden time for the provider survey is estimated at 30 minutes, with a range of 25 to 35 minutes (see Exhibit 2).
Exhibit 2. Cross-Site Data Collection Burden for Client Survey
Instrument/Activity |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Hour Burden |
Hourly Wage |
Total Costa |
Client survey |
7,329 |
1 |
7,329 |
0.15 |
1,099 |
$18.84 |
$20,712 |
Provider survey |
4,083 |
1 |
4,083 |
0.50 |
2,042 |
$27.00 |
$55,121 |
TOTAL |
11,412 |
|
11,412 |
|
3,141 |
|
$75,833 |
a Total respondent cost is calculated as hourly wage × time spent on survey × number of respondents.
Estimate the annualized cost burden to the respondent for the collection of information from clients. As data collection will be completed within 1 year, the following estimates represent the total cost burden for the client survey. There are no direct costs to respondents other than their time to participate in the study. The estimated total cost of the time respondents spend completing these surveys is $20,712 (number of total client respondent hours × $18.84, the estimated average hourly wages for adults as published by the Bureau of Labor Statistics, 2006).
Estimate the annualized cost burden to the respondent for the collection of information from providers. As data collection will be completed within 1 year, the following estimates represent the total cost burden for the provider survey. There are no direct costs to respondents other than their time to participate in the study. The estimated total cost of the time respondents spend completing these surveys is $55,121 (number of provider respondent hours × $27, the estimated average hourly wages for individuals working in community and social service management as published by the Bureau of Labor Statistics, 2006).
There are no respondent costs for capital or start-up or for operation or maintenance.
The estimated cost to the government for the cross-site data collection is $4,544,575 (Exhibit 3). This includes $4,515,175 for a 3-year contract, including sampling, site visits, survey implementation, data collection, processing, and reports, and approximately $9,800 of Task Order Officer’s salary spent on oversight and analysis per year represents SAMHSA costs to manage/administer the survey for 10% of one employee (GS–14). The total annualized cost is approximately $1,514,858.
Exhibit 3. Estimates of Annualized Cost to Government
Activity/Employee |
Total Cost |
Annualized Cost |
ATR cross-site evaluation |
$4,515,175 |
$1,505,058 |
SAMHSA Employee (GS-14, at 10%) |
$29,400 |
$9,800 |
Totals: |
$4,544,575 |
$1,514,858 |
This is a new collection of information.
Exhibit 4 outlines the key time points for the study and for the collection of information. The requested period also allows for training and start-up activities associated with the preparation for data collection.
Exhibit 4. Time Schedule for Entire Project
Activity |
Time Schedule |
Evaluation Design Questions submitted |
November 2007 |
Expert Panel |
February 2008 |
Obtaining OMB approval for data collection |
Spring 2009 |
Data collection begun |
1 month post OMB approval |
Data collection complete |
11 months post OMB approval |
Full data analysis |
Beginning 11 months post OMB approval |
Dissemination of findings |
Beginning 13 months post OMB approval through September 2010 |
The ATR cross-site evaluation is designed to produce knowledge about the impact of the ATR Program as implemented across the 24 grantees. It is therefore important to prepare and disseminate reports, concept papers, documents, and oral presentations that clearly and concisely present project results so they can be understood by both technical and nontechnical audiences. The cross-site evaluation team will
produce rapid-turnaround analysis papers, briefs, and reports;
prepare and submit monthly progress reports;
prepare a final ATR cross-site findings report, including an executive summary;
deliver presentations at professional and federally sponsored conventions and meetings; and
disseminate reports and materials to entities inside and outside CSAT.
The analysis centers on specific provider- and client-level questions found in Attachment 10. Potential analyses on the client and provider survey data used to address these questions are detailed in the remainder of this section.
The client survey analysis will include descriptive statistics and model-based analyses. When supported by the data, analyses may be performed on the total sample pooled across grantees and on subsets of grantees as appropriate.
Descriptive statistics will focus primarily on means, proportions, and tests of statistical significance of difference. Tables will report key statistics such as means, sample size, standard errors, and t- and χ2- test results where appropriate. Findings may also be presented separately for key characteristics of the grantee (e.g., centralized versus decentralized screening and assessment program models), provider (e.g., clinical treatment provider versus RSS), and client (e.g., male versus female) as appropriate. Design effects will be taken into account, as appropriate, in computing means or proportions, their associated standard errors, and for related bivariate tests. Attachment 11 provides illustrative sample table shells in which the descriptive analyses may be reported.
The model-based analyses will account for the hierarchical nature of the data (clustering of clients within providers within grantees) and the longitudinal nature of the data (e.g., repeated measures of GPRA) where appropriate. A variety of methods are available to account for clustering and longitudinal data, but in most cases we propose to use generalized linear mixed models (GLMM) (McCulloch, 2003), an extension of the generalized linear model in which the linear predictor contains random effects in addition to the usual fixed effects. The quantitative model-based analyses will examine several different outcome variables of interest (e.g., satisfaction with program; days of drug use; criminal justice status, employment) and these variables may include dichotomous, categorical, or continuous measures, all of which can be handled within the GLMM framework. Examples of key explanatory variables in the models are client demographic characteristics (e.g., gender, race/ethnicity), client program experience (e.g., services received, satisfaction), grantee characteristics (e.g., geographic diversity of program, screening and assessment protocols), and provider characteristics (e.g., services offered, faith-based versus secular provider, urban versus rural location). Attachment 11 includes an illustrative sample table shell in which the model-based analysis results may be reported.
The provider analyses will be based primarily on descriptive statistics on service type (i.e., clinical treatment and RSS) and provider characteristics, experiences, and attitudes. Tables will report key statistics, such as means, sample size, standard errors, and t- and χ2- test results where appropriate. The basic approach will pool data across grantees. When appropriate, findings will be presented separately for key provider characteristics (e.g., urban versus rural, utilized [i.e., actually served ATR clients] versus non-utilized). In addition, statistical modeling, similar to that described for the client data analysis, may be used if supported by the data. Attachment 12 is a table shell in which results of the analysis of provider data may be reported.
The expiration date for OMB approval will be displayed on all data collection instruments for which approval is being sought.
There are no exceptions to the certification statement. The certifications are included in this submission.
The targeted universe for the ATR client survey are individuals who have or are receiving ATR-funded services and are eligible for a 6-month post-intake GPRA follow-up across the 24 funded grantee programs. ATR grantees are mandated to collect GPRA follow-ups on at least 80% of such clients. Eligibility for ATR funding varies by grantee program as each grantee designed its respective ATR Program to best address local needs. The client survey will be administered once to a sample of ATR clients approximately 5 to 8 months post-intake to coincide with the GPRA follow-up.
The client survey sampling frame will be based on the 6-month post-intake GPRA follow-up occurring during a selected 2- to 4-month time period. The sampling frame will include a census of all ATR clients receiving a 6-month post-intake GPRA during this period. Because comprehensive data are not yet available on client flow for the current ATR Program being evaluated, the cross-site evaluation team estimated expected population statistics and client flow from GPRA data collected by grantees during the first ATR Program (2004–2007) and current grantee client goals mandated by SAMHSA/CSAT. Based on these estimates, client surveys will be distributed within a given grantee to all clients receiving a GPRA follow-up over a designated 2- to 4-month period. During this period, client flow is expected to be representative (based on estimated population statistics) of the ATR client population across the following key domains: gender, race/ethnicity (non-Hispanic white, non-Hispanic black, non-Hispanic other, Hispanic) and methamphetamine use. Incoming ATR GPRA data will be monitored to ensure that a representative sample is being obtained and that the data collection period is adjusted as needed to minimize client burden and data collection efforts. During the data collection period, the maximum projected number of eligible clients is 9,165 of which 7,329 (80%) are expected to respond.
Substance abuse service provider organizations (e.g., residential substance abuse treatment facilities, organizations hosting Alcoholics Anonymous meetings, individual substance abuse treatment practitioners) participating in a grantee’s ATR provider network are eligible for the ATR provider survey. At each provider organization, the survey targets a key informant, typically a director or manager within the organization most knowledgeable on the ATR Program. The sampling frame will be a census of all providers participating in a grantee’s ATR network as of December 31, 2008. As of this date the sampling frame will be set (i.e., “frozen”), and any providers joining a grantee’s ATR provider network after this date will not be eligible for the survey.
During the data collection period, the maximum projected number of eligible provider organizations is 5,103 of which 4,083 (80%) are expected to respond.
Client surveys among the GPRA follow-up clients within a given grantee program will be collected over a 2- to 4-month period. Based on projected client flow and ATR-1 historical data, a set time period for data collection at each grantee is expected to obtain a representative sample and support pair-wise comparisons over the following key domains: gender, race/ethnicity, and methamphetamine use. Exhibit 5 presents grantee-level information on expected volume of patient flow during the proposed data collection period.
Exhibit 5. Grantee Client Flow
Grantee |
2-Month Client Flow |
Respondent Sample |
|
Adult |
Adolescent |
||
AK Southcentral Foundation |
54 |
43 |
0 |
AZ |
75 |
60 |
0 |
CA |
426 |
70 |
270 |
Cherokee Nation (OK) |
158 |
116 |
11 |
CO |
370 |
270 |
26 |
California Rural Indian Health Board (CRIHB) |
426 |
307 |
33 |
CT |
573 |
458 |
0 |
DC |
491 |
393 |
0 |
HI |
104 |
76 |
7 |
IA |
278 |
222 |
0 |
IL |
382 |
306 |
0 |
IN |
587 |
429 |
41 |
|
|
Cont. |
|
Cont |
|
|
|
Grantee |
2-Month Client Flow |
Respondent Sample |
|
Adult |
Adolescent |
||
|
|
|
|
Inter-Tribal Council (ITC) MI |
322 |
235 |
22 |
LA |
344 |
242 |
34 |
MO |
2,326 |
1,854 |
7 |
MT-WY Tribal Leaders Council (TLC) |
158 |
116 |
11 |
NM |
118 |
94 |
0 |
OH |
216 |
173 |
0 |
OK |
341 |
273 |
0 |
RI |
85 |
62 |
6 |
TN |
235 |
188 |
0 |
TX |
210 |
157 |
10 |
WA |
382 |
279 |
26 |
WI |
502 |
402 |
0 |
Total |
9,165 |
6,825 |
504 |
Volume is based on monthly average data provided by GPRA, ATR-1 based estimates, ATR-2 proposed client flows, and length of data collection.
The sampling design will permit data to be pooled across all 24 grantees across each key domain: gender, race/ethnicity (non-Hispanic white, non-Hispanic black, non-Hispanic other, and Hispanic) and identified methamphetamine users. Exhibit 6 presents the expected number of adult respondents for each analysis domain by grantee.
Exhibit 6. Expected Respondent Sample by Grantee and ATR domain
|
Adult Distribution |
||||||
Grantee |
Male |
Female |
NH White |
NH Black |
NH Other |
Hispanic |
Meth |
AK SCF |
30 |
13 |
20 |
12 |
6 |
5 |
2 |
AZ |
42 |
18 |
28 |
17 |
8 |
7 |
36 |
CA |
47 |
23 |
6 |
9 |
36 |
18 |
14 |
Cherokee Nation (OK) |
80 |
36 |
54 |
33 |
16 |
13 |
7 |
CO |
187 |
83 |
125 |
77 |
36 |
31 |
16 |
CRIHB |
152 |
155 |
6 |
0 |
280 |
21 |
38 |
CT |
318 |
140 |
213 |
131 |
61 |
53 |
0 |
DC |
272 |
121 |
182 |
113 |
53 |
45 |
47 |
HI |
53 |
23 |
35 |
22 |
10 |
9 |
5 |
IA |
154 |
68 |
103 |
64 |
30 |
25 |
73 |
IL |
240 |
66 |
43 |
233 |
9 |
21 |
27 |
IN |
297 |
132 |
199 |
123 |
58 |
49 |
25 |
ITC-MI |
163 |
72 |
109 |
67 |
32 |
27 |
14 |
LA |
160 |
82 |
119 |
95 |
22 |
5 |
12 |
MO |
1,391 |
463 |
414 |
528 |
883 |
29 |
147 |
MT-WY TLC |
80 |
36 |
54 |
33 |
16 |
13 |
7 |
NM |
66 |
28 |
21 |
3 |
26 |
44 |
28 |
|
|
|
|
|
|
|
Cont. |
|
|
||||||
Cont. |
|
||||||
|
Adult Distribution |
||||||
Grantee |
Male |
Female |
NH White |
NH Black |
NH Other |
Hispanic |
Meth |
OH |
120 |
53 |
80 |
50 |
23 |
20 |
3 |
OK |
189 |
84 |
127 |
78 |
37 |
31 |
181 |
RI |
43 |
19 |
29 |
18 |
8 |
7 |
4 |
TN |
123 |
65 |
103 |
48 |
35 |
1 |
25 |
TX |
94 |
63 |
48 |
33 |
5 |
71 |
5 |
WA |
193 |
86 |
130 |
80 |
37 |
32 |
17 |
WI |
271 |
131 |
98 |
236 |
28 |
39 |
26 |
Total |
4,765 |
2,060 |
2,346 |
2,103 |
1,755 |
616 |
759 |
Notes: NH = non-Hispanic; meth = methamphetamine users
Volume is based on monthly average data provided by GPRA, ATR-1 based estimates, ATR-2 proposed client flows, and length of data collection. Total exceeds 9,165 due to domain overlap.
The ATR evaluation team will monitor client flow over these six domains through GPRA data and focus data collection efforts to ensure that a sufficient number of clients in each domain are sampled. Once a sufficient number of clients per domain have been sampled, data collection efforts targeting that population will cease. This will reduce client burden and costs.
The proposed sampling paradigm is designed to obtain samples large enough within each domain of interest (n=388) to ensure sufficient power (80%) and confidence (95%) for conducting two-sample t-tests for detecting differences of + 10% from an estimated 50% dichotomous outcome measure. Due to low expected client flow among adolescents, analyses among this subpopulation will be limited. Some increase in the variance estimation is expected due to unequal weighting effects caused by potential disproportionate response rates within grantees and analysis domains, as well as by aspects related to GPRA follow-up procedures. However, this impact on the overall design effect is expected to be minimal, and the sampling design ensures adequate power across all adult analytic domains for a design effect as large as 1.5. Additionally, effective sample sizes among adults are maintained within all domains except Hispanics and identified methamphetamine users for design effects as high as 4.5. Estimation and significance testing will be conducted using SAS and Stata software as appropriate, which can properly account for unequal weighting effects, as well as any possible probability-based sampling schemes that may have been implemented by grantees for GPRA follow-up.
Data collection procedures for the client survey will accommodate grantee variability of grantee GPRA follow-up procedures. The survey will be administered in-person as a self-report questionnaire. The cross-site evaluation team will work with ATR grantees to administer the client survey. Each grantee has local staff (i.e., the local GPRA administrator) responsible for administering the GPRA follow-up. These staff will help distribute the self-report client survey.
Specifically the self-report questionnaire procedures are:
The local GPRA administrator briefly mentions the survey prior to administrating the GPRA follow-up.
The local GPRA administrator will ask the client after the GPRA follow-up session if they would like to participate in the survey as directed by the client survey script provided by the cross-site evaluation team (see Attachment 6.1).
If the client agrees, the local survey administrator will obtain verbal informed consent and will give the client a written informed consent form and a frequently-asked-questions form as directed in the client survey script (see Attachments 6.3 and 6.4).
Following client consent, the local GPRA administrator records the client’s GPRA ID number on the client survey form and provides the client with the client survey and a tamper evident return envelope (see Attachment 3).
If the local GPRA administrator does not have access to the client’s GPRA ID, an alternative ID number, such as a grantee-generated dummy ID number that is not a personal identifier, will be recorded on the survey. The ATR grantee will provide the cross-site evaluation team with a crosswalk database linking the alternative ID number to the client’s GPRA ID number. This crosswalk will be maintained on a secure server maintained by the contractor.
While unlikely, if the local GPRA administrator uses a personal identifier or name as an alternative ID, a tear-away facepage will added to the client survey (see Attachment 6.6). The cross-site evaluation team will print a randomly generated dummy ID on the facepage and the client survey. The local GPRA administrator will record the alternative ID on the facepage and deliver it separately to the ATR grantee representative. The grantee representative will deliver a crosswalk database between the randomly generated IDs and the client’s GPRA ID to the cross-site evaluation team. At no time will the cross-site evaluation team receive clients’ personal identifiers or names.
Respondent completes survey without the local GPRA administrator’s assistance.
Respondent seals completed survey in the provided tamper evident envelope and returns it to the local GPRA administrator.
The local GPRA administrator gives the incentive to the respondent.
The local GPRA administrator immediately mails the completed survey directly to the cross-site evaluation team via the U.S. postal service.
In some cases, the local GPRA administrator will return the sealed surveys to the ATR grantee. The ATR grantee will then return the sealed surveys in bulk once a week to the cross-site evaluation team via provided Federal Express mailers.
All returned client survey forms (including the special case described below) will be received by the contractor’s Survey Support Division, which will provide secure storage for the survey forms and perform all data entry tasks. Data will be optically scanned (e.g., TeleForm) and loaded in a database stored on a secure server maintained by the contractor. Access will be limited to cross-site evaluation team members involved in data collection and analysis.
In special cases, some 6-month GPRA follow-ups may be administered over the telephone. In these instances, the ATR client survey will be administered using the following procedures:
The local GPRA administrator briefly mentions the survey prior to administrating the GPRA follow-up.
The local GPRA administrator will ask the client after the GPRA follow-up session if they would like to participate in the survey (see Attachment 6.2).
If the client agrees, the local survey administrator will obtain verbal informed consent.
Following client consents, the local GPRA administrator will record a dummy ID (e.g., a randomly assigned number, a local non-personal client ID number) and the client’s GPRA ID on a tear-away facepage (see Attachment 6.6). The dummy ID will be recorded on the survey form and the return mailer. Once the completed survey is returned, the tear-away facepage will be used to link the client survey to the ATR GPRA data. This procedure will ensure that a client’s GPRA ID and name are not linked.
The local GPRA administrator will mail the client a survey packet; the survey packet includes all documentation provided for the in-person interview: a written informed consent form, a frequently-asked-questions document, the client survey, mailing instructions for returning the completed survey, and a prepaid return envelope (see Attachments 6.3, 6.4, 3, and 6.5).
Respondent completes the survey and returns it to the local GPRA administrator, who mails the client a $10 incentive.
The local GPRA administrator forwards the completed, sealed survey to the cross-site evaluation team.
The cross-site evaluation team will obtain publicly available provider contact information from ATR grantees. This information will include an organization’s address, telephone number, and ATR-specific information such as types of services provided and the name and e-mail address of a key informant at each provider. A randomly generated ID number will be assigned to each provider that will link provider contact information with the provider survey data. The cross-site evaluation team will create unique, secure user accounts for each provider organization and will contact each provider via e-mail, first with a letter introducing the survey, and later with details on how to access and log in to the Web survey (see Attachments 7.1 and 7.3). After logging in to the provider survey Web site, providers will be asked to review the informed consent prior to beginning the provider survey (see Attachments 7.2 and 5). A provider organization’s user name will be the randomly generated ID number linked to the provider’s contact information. A secure password will be e-mailed (or mailed if the provider organization lacks a secure e-mail address) separately from the user name.
Provider organizations will be given a set period of time to complete surveys and will be sent up to two reminder letters or e-mails to maintain response rates (see Attachment 7.4). Additional reminders and promotional efforts may be carried out by the grantees. An example grantee provider survey information document is attached (see Attachment 7.7).
Once a provider submits a completed Web survey, the cross-site evaluation team will be electronically notified, and the data will be automatically stored on a secure, password protected server maintained by the contractor (additional details on the standard Web site security used here are provided on page 11). Access to the database will be limited to evaluation project staff directly involved in data collection and analysis. The data will include no personal identifiers for the individual completing the survey. To ensure respondent privacy, the cross-site evaluation team will maintain the provider organization contact lists obtained from grantees separately from the provider survey data. A provider’s contact information and completed survey data will be linked by the randomly generated ID number. A crosswalk database will be created from the provider contact list and will only include the provider’s random ID number, zip code, city and state/grantee. Only select cross-site evaluation team members will have access to both the crosswalk database and the provider contact information. This database will be linked to the provider survey data via the random ID number. Contact lists and all provider survey data will be stored separately on the contractor’s servers. Once data collection is complete, all contact information will be purged.
Providers unable to access the Web survey or without e-mail addresses will be provided with a mailed paper-and-pencil version of the survey. Survey packets will be mailed by Federal Express and returned directly to the cross-site evaluation team via a return Federal Express mailer.
A survey packet will include a cover letter with specific mail survey introduction, instructions, and a copy of the informed consent form (see Attachments 7.2 and 7.5). Once the specified period of time to complete the surveys lapses, providers who have not yet responded will be sent up to two reminder letters to maintain response rates (see Attachment 7.6). The contractor’s Survey Support Division will receive the mailed provider survey forms and perform data entry tasks (e.g., TeleForm or manual data entry). Mailed surveys will be stored in secure storage and only accessible to select cross-site evaluation team members. When data collection is complete, the survey forms will be destroyed.
The cross-site evaluation team expects the client survey to have a response rate of 80%. During the data collection period, the maximum projected number of eligible clients is 9,165 of which 7,329 (80%) are expected to respond. The general approach to recruitment and maintaining the expected response rates is as follows:
Independently of the client survey process, local GPRA administrators are expected to obtain detailed contact information from each client to facilitate the 6-month post-intake GPRA follow-up.
The cross-site evaluation team will develop concise survey distribution instructions for grantee and their local GPRA administrators to ensure participants rights while minimizing burden for the ATR grantees.
The cross-site evaluation team has developed a concise client survey instrument.
The cross-site evaluation team will offer surveys in English and Spanish languages if consistent with a grantee’s GPRA data collection protocols.
The cross-site evaluation team will offer an incentive for survey completion.
Clients will be provided an incentive (e.g., a $10 gift card) upon completing the survey. And incentive will be given with the ATR client survey because the target population is a hard to reach population and an incentive is expected to significantly increase response rates. The incentive size balances the expected impact on responses rates with the estimated low burden time (estimated at 6 to 10 minutes). Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. This research is detailed in Section A.9, page 8.
The cross-site evaluation team expects the provider survey to have a response rate of 80%. During the data collection period, the maximum projected number of eligible provider organizations is 5,103 of which 4,083 (80%) are expected to respond. To maximize initial response rates for the provider survey, the cross-site evaluation team members will follow protocols to ensure that all eligible providers are aware of the survey and its importance to the ATR cross-site evaluation. The cross-site evaluation team will send lead letters and clear instructions to providers to solicit provider participation in the survey. Grantees will be asked to inform providers of the voluntary survey at local providers meetings and through regular communication to raise awareness of the survey and its importance to the ATR evaluation (see Attachment 7.7).
Cross-site evaluation team members will also send reminder letters to providers not completing the survey within an established time frame. The follow-up procedures include sending up to two reminder letters via e-mail or mail (see Attachment 7.4).
Using the Web as the survey medium will facilitate survey completion by making the survey convenient and readily accessible to providers. When a provider is unable to access the Web instrument, survey staff will provide a paper-and-pencil version via mail (e.g., Federal Express). The provider survey will include a toll-free number and an e-mail address to allow provider organization staff to contact the cross-site evaluation team directly if they have any questions about the survey or their rights as a participant.
The cross-site evaluation team tested the client survey with five test respondents and found that the client survey takes approximately 6 minutes to complete. In addition, it takes approximately 3 minutes to complete the informed consent procedures. The survey, including the informed consent procedures, took between 6 and 11 minutes to complete.
The client survey contains items from the Mental Health Statistics Improvement Program (MHSIP-28) Consumer Survey to measure client satisfaction. In addition, questions on client choice and experience have been developed based on a review of grantee client surveys; various established data instruments drawn form the literature, and expertise of the cross-site evaluation team.
The cross-site evaluation team tested a pencil-and-paper version of the provider survey with three test respondents and found that it takes approximately 20 minutes to complete. In addition, it takes 10 minutes to read the introductory letter, the informed consent, and login instructions for a total of 30 minutes on average. Depending on a provider organization’s experience with ATR, burden time is expected to range from 25 to 35 minutes.
The provider survey was developed to specifically address the unique provider organization experiences associated with participation in a grantee’s ATR network. Survey questions were based on the cross-site evaluation provider-level questions and developed by experts on the ATR cross-site evaluation team.
The evaluation team will thoroughly test the Web instruments and all supporting computing systems before beginning data collection. In addition, data storage, retrieval procedures, and all supporting systems will be tested prior to data collection.
As noted in Section A.8, pages 7–8, the cross-site evaluation team has consulted extensively with an expert panel that has reviewed data collection and analysis methodologies outlined in this package. In addition, the cross-site evaluation team includes qualified staff who are being consulted throughout the project on various statistical aspects of the design, methodological issues, economic analysis, database management, and data analysis. Exhibit 7 lists these advisors.
Exhibit 7. Key Cross-Site Evaluation Team Members and Senior Advisors
Expert |
Affiliation |
Contact Information |
|
Deepa Avula Task Order Officer |
Substance Abuse and Mental health Services Administration Center for Substance Abuse Treatment 1 Cherry Choke Road Rockville, MD 20857 |
Phone: 240-276-2961 |
|
Laura J. Dunlap,
PhD |
Program Director Rockville, MD 20852 |
Phone: 301–816–4622 |
|
|
|
Cont. |
|
Cont. |
|
|
|
Expert |
Affiliation |
Contact Information |
|
Michael A. Penne, MPH |
Statistician |
Phone: 919–541–5988 |
|
Paul Levy, PhD
|
Senior Fellow, Statistics in the Health Sciences RTI
International |
Phone: 919–316–3857 |
|
Lisa Carley–Baxter, MA
|
Senior Survey Methodologist Survey Research Division RTI
International |
Phone: 919–485–2616 Fax: 919–316–3866
|
|
Jeremy W. Bray, PhD
|
Fellow, Health
Economist |
Phone: 919–541–7003 |
|
Georgia T. Karuntzos, PhD |
Associate
Director |
Phone: 919–541–6159 |
|
Carol L. Council, MSPH |
Senior Research
Associate |
Phone: 919–485–5722 |
|
Dominic Hodgkin, PhD
|
Senior Researcher Schneider Institute For Health Policy Heller School for Social Policy and Management Brandeis University, MS 35 415 South Street, Waltham, MA 02454–9110 |
Phone: 781–736–8551 Fax: 781–736–3928 E-mail: [email protected] |
|
Minakshi Tikoo, PhD
|
Senior Researcher University of Connecticut Health Center, Dept. of Community Medicine and Health Care Farmington, CT, 06030–1910 |
Phone: 860–679–5559 Fax: 860–679–5464 E-mail: [email protected] |
|
Rick Offner, PhD
|
Senior Researcher Danya International 8737 Colesville Road, Suite 1200 Silver Spring, MD 20910 |
Phone: 301–565–2142 Fax: 301–565–3710 E-mail: [email protected] |
|
McCulloch, CE. (2003). Generalized Linear Mixed Models. NSF-CBMS Regional Conference Series in Probability and Statistics Volume 7. Institute of Mathematical Statistics (IMS): USA.
Wright, D., Bowman, K., Butler, D., & Eyerman, J. (2002). Nonresponse Bias from the National Household Survey on Drug Abuse Incentive Experiment. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, St. Petersburg Beach, FL.
Attachment 1: ATR Request for Applications, Section 2
Attachment 2: Referenced ATR PART Review
Attachment 3: Client Survey
Attachment 4: Provider Survey
Attachment 5: Provider Survey Screen Shots
Attachment 6: Client Survey Supporting Documents
Attachment 7: Provider Survey Supporting Documents
Attachment 8: Justification for Waiver of Parental Consent
Attachment 9: Network Security at RTI International
Attachment 10: Process and Outcome Evaluation Questions, Provider and Client Levels
Attachment 11: Table Shells—Descriptive Results and Model Based
Attachment 12: Table Shell—Descriptive Results
1CSAT initially funded 15 grantees in the first ATR Program from 2004–2007 (ATR-1). In 2007, CSAT funded 24 grantees (including 11 grantees from the ATR-1 program) for the second ATR Program from 2007-2010 (ATR-2).
2 The ATR Expert Panel was held in February 2008 and included external experts and grantee representatives. Section 8 discusses the Expert Panel in more detail.
| File Type | application/msword |
| File Title | ACCESS TO RECOVERY (ATR) PROGRAM CROSS-SITE EVALUATION |
| Author | Cassie Williams |
| Last Modified By | sorme |
| File Modified | 2008-10-15 |
| File Created | 2008-10-14 |