OMB Supporting Statement National AAFT Evaluation_revised_FINAL DRAFT

Evaluation for Assertive Adolescent & Family Treatment (AAFT) Program

SUPPORTING STATEMENT

A. Justification

1) Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) is requesting approval from the Office of Management and Budget (OMB) for the three versions of the AAFT Implementation Survey.

• Principal Investigators/Project Directors

• Clinical Supervisors/Clinicians

• Evaluators/Data Managers

The AAFT Implementation Survey will be used as the primary method for measuring the readiness of AAFT3 grantees to implement the project as well as collect data implementation process throughout the life of the grant. More specifically, the survey will contain questions about the background, experience, and skills of service providers and administrators at each site, as well as practices and behaviors that may or may not change over time as a result of the training and technical assistance offered through this grant.

The AAFT program was initiated in funding year 2006 to provide evidence-based substance abuse services to adolescents and their families, as well as to transition-age youth, caregivers, and their families/mentors. This program is based on evidence that families/primary caregivers are an integral part of the treatment process and their inclusion increases the likelihood of successful treatment and reintegration of adolescents and transition-age youth into their communities following formalized treatment. There have been three cohorts of grantees funded under this program since 2006—the most recent cohort of 14 grantees was funded in 2009.

In supporting AAFT, to ensure that the program is implemented with fidelity, CSAT has provided, through Chestnut Health Systems, a package of implementation supports, including:

• Manual-assisted training that guides and provides measures of fidelity for A-CRA and ACC

• Certification for clinical and supervisory staff in the A-CRA/ACC models

• Training/certification in GAIN for assessment and clinical interpretation

• The use of a Web-based tool used to monitor service delivery, fidelity to the treatment model, clinical/supervisory certification, and data management

• Monitoring, coaching, mentoring, and support for clinicians, supervisors, and data managers

• Implementation calls paired with monthly implementation progress reports

• Topical workgroups that share ideas and resources (e.g., Cultural Responsiveness Committee).

AAFT requires grantees to implement the Adolescent Community Reinforcement Approach (A-CRA) coupled with Assertive Continuing Care (ACC) to provide treatment that is context-specific, family-centered, and community-based. Grantees are also required to use the Global Appraisal of Individual Needs (GAIN) as the common assessment instrument across programs to improve intake assessment, clinical interpretation, monitoring, and data management. Clinical staff must participate in training and certification processes for the clinical interventions and assessment instrument as well as in each of the previously listed AAFT components.

Advocates for Human Potential’s (AHP) role with respect to the SAMHSA/CSAT AAFT grantees

AHP was funded in September 2009 to conduct the cross-site evaluation of the Assertive Adolescent and Family Treatment (AAFT) program’s third grantee cohort. The overarching objective of the cross-site evaluation process and outcome evaluation is to assess and document the process of implementation in the 2009 cohort of AAFT grantees and to explore the role that implementation supports play in how well these programs evolve. Grantees must evaluate the process and implementation of their local programs as well as participate in the cross-site evaluation regarding the implementation and impact of the program described later in this document. The AAFT initiative provides a wealth of high quality data. Therefore, we intend to make maximum possible use of these data sources and are proposing new data collection only where we believe existing resources cannot be used to address the evaluation questions. The cross-site evaluation, will utilize one additional data collection instrument: the AAFT Implementation Survey to gather data from a range of grantee personnel to evaluate the implementation, expansion, and sustainability of adolescent substance use services developed under the AAFT program. As part of receiving funding, grantees are required to cooperate with a cross-site evaluation. This survey is being conducted as part of the AAFT cross-site evaluation.

2. Purpose and Use of Information

The purpose of the AAFT Implementation Survey is to gather longitudinal data, at the end of each project year, from program administrators, clinical staff, and evaluators regarding their experiences implementing the AAFT project.

The AAFT Implementation Survey presents a common framework for capturing program implementation across the AAFT program grantees. The tool has three versions, tailored to address the respondents’ roles in the grant (Principal Investigators/Program Directors, Clinical Supervisors/Clinician, and Evaluators/Data Managers), and measures a range of domains using mostly close-ended questions, with some open-ended responses. While some data elements are from existing instruments (e.g., Organizational Readiness for Change, ORC), original question sets were also developed to capture other constructs such as reactions to Chestnut supports and perceived implementation of AAFT components. The following is a brief description of the domains and types of information requested in each section.

1. Background and experience. The purpose of this section is to provide a basic overview of the educational background and clinical experience of grantee personnel at the administrative, clinical, and evaluator levels. Respondents are asked to indentify their primary role on the project, educational degrees/certifications/licenses, years of experience in the substance use treatment field, prior experience with the treatment population, evidence-based practices, GAIN, and A-CRA/ACC.

2. Participation in and reactions to AAFT components. Several sections of the survey ask respondents to indicate their level of participation in and reactions to the variety of implementation supports provided as part of the AAFT program. The purpose of these sections is to document attitudes toward AAFT model components (e.g., manualized treatment, semi-structured interviewing, certification processes), involvement with and reactions to the components, organizational changes made to incorporate components of the AAFT program, and perceived usefulness of support services.

3. Adaptations/modifications to the model. The purpose of this section is gain a better understanding of adaptations or modifications to the A-CRA/ACC treatment model at grantee sites. Respondents are asked to indicate if their program has made any modifications to their A-CRA/ACC treatment model and, if so, describe the modifications made. Additionally, they are asked to rate how well they believe these modifications have worked at their site.

4. Barriers to implementation. The purpose of these items is to identify barriers encountered in implementation and compensatory strategies. Respondents are asked to rank the difficulty of overcoming certain components of the AAFT program. The components include those related to: program/organizational issues, service delivery, grant required activities, and research/evaluation activities. For the three most challenging barriers, grantees are asked to describe the impact on their site and strategies utilized to overcome these barriers.

5. Staff turnover. The purpose of this section is to document staff turnover within their grant-funded program and record the effect of staff turnover on the grant program. Respondents are asked to indicate if, for several personnel levels (e.g., administrative, clinical, evaluation/research), there has been new staff hired during the prior project year. If so, they are also asked to rate their perception of the overall effect the change has had on their grant program as well as to describe the effects of the change.

6. Barriers to service delivery. The purpose of this section is to determine what barriers adolescents/families may have experienced in receiving services prior to the AAFT grant program and after implementation of the AAFT grant. Among the list of typical barriers to services respondents are asked to rate are: transportation, child care, need for mental health treatment, neighborhood safety, affordability of treatment, lack of family involvement, language/cultural issues, and general resistance to treatment related to readiness to change, stigma, and shame.

7. Readiness for change. The purpose of this section is to gather information on organizational-level and staff-level readiness for change and attitudes regarding substance use treatment service delivery and disorders. Respondents are asked to rate themselves and their organizational climate in several subject areas: motivation for change, adequacy of resources, training needs, efficacy, and adaptability (using subscales from the Organizational Readiness for Change, ORC); practitioner attitudes toward the use of treatment manuals (items from Addis and Krasnow’s questionnaire); opinions regarding substance use treatment and disorders, attitudes toward program evaluation/research, and the use of data (original sets of questions developed from review of literature).

8. Sustainability. The purpose of this section is to document any plans the grantee site may have to sustain the program after CSAT funding has ended. Respondents are asked to indicate if they are engaged in any activities toward sustainability of the AAFT program. If so, they are asked to describe the activities and comment on which components of the AAFT program are likely to continue and why.

9. Lessons learned and accomplishments. The purpose of this section is to document any lessons learned as a result of their involvement in the AAFT program as well as any accomplishments and quantify them, if possible. Among these items respondents are asked to: reflect on the prior year and consider what they might have done differently with the knowledge they’ve obtained, what advice they would offer to other organizations intending to apply for this

SAMHSA/CSAT will use the AAFT program’s information from the AAFT Implementation Survey to report on:

• Practices and strategies used in the grantee programs

• Contextual factors that influence project implementation

• Fidelity to the AAFT program

• Organizational and service provider changes in attitudes, skills, and practices

• Challenges, accomplishments, and lessons learned

• Program sustainability and expansion of the program.

3) Use of Information Technology

The AAFT Implementation Survey will be available to grantees as a Web-based survey. Once finalized and approved by OMB, the IT Team will design and manage and interactive, 508-compliant Web site that will allow grantees to submit the survey online.

4) Efforts to Identify Duplication

This data collection is significant only to specific components of this program and is not collected anywhere else.

5) Involvement of Small Entities

There is no significant impact on small entities or small businesses.

6) Consequences if Information Collected Less Frequently

The information for the AAFT Implementation Survey will be collected once a year—at the end of each project year. If it was collected less frequently, there may be changes that occur in the grantees’ work or contextual changes that might not be captured because of the length of time between the event and the reporting period.

7) Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with 5 CFR 1320.5(d)(2).

8) Consultation Outside the Agency

The Federal Register Notice required by 5 CFR 1320.8(d) soliciting comments on the information was published on August 2, 2010, Vol. 75, p. 45124. There were no public comments received.

An Evaluation Advisory Workgroup (EAW) was established for the cross-site evaluation data collection instruments. Feedback on the AAFT Implementation Survey was solicited from the SAMHSA/CSAT TOO, AHP, Chestnut Health Systems (the current TA contractor to the program), expert consultants, and grantee participants in the EAW (including evaluators, clinical supervisors/clinicians, and data managers). Based on the feedback from the EAW and others mentioned above, modifications were made to the instruments. On the monthly grantee conference call on March 18, 2009, details regarding the cross-site evaluation, the EAW, and the AAFT Implementation Survey were discussed. Annual meetings between grantees, federal project officers and project staff will review evaluation components and collection efforts.

Organizations and individuals that reviewed the AAFT Implementation Survey include the following:

Name/Title	Address	Contact Information
Terri Tobin, Ph.D. Evaluation Director	Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776	Phone: (978) 261-1418 [email protected]
Karl D. Maxwell, Task Order Officer	Center for Substance Abuse Treatment, SAMHSA 1 Choke Cherry Road Room 5-1062 Rockville, MD 20857	Phone: (240) 276-2824 [email protected]
Amy Salomon, Ph.D. Director of Research & Evaluation	Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776	Phone: (978) 261-1409 [email protected]
Nick Huntington, M.A. Senior Research Associate	Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776	Phone: (978) 261-1453 [email protected]
Denise Lang, BS Research Associate	Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776	Phone: (401) 323-9678 [email protected]
Chestnut Health Systems	448 Wylie Drive Normal, IL 61761	Phone: (309) 451-7801
Expert consultants
Ken Winters, Ph.D.	University of Minnesota Dept. of Psychiatry F282/2A West 2450 Riverside Avenue Minneapolis, MN 55454	Phone: (612) 273-9800 [email protected]
Charles Turner, Ph.D.	Oregon Research Institute 1715 Franklin Blvd. Eugene, OR 97403-1983	Phone: 541-484-2123 [email protected]
Heather Gotham, Ph.D.	Mid-America Addiction Tech Transfer Center University of Missouri-Kansas City 5100 Rockhill Road Kansas City, MO 64110-2499	Phone: (816) 482-1135 [email protected]

Evaluation Advisory Workgroup
Southern California Alcohol & Drug Programs (SCADP)—Youth & Family Services	8700 Cleta St. Downey, CA 90241	Phone: (562) 862-9766
Child and Family Services	99 Hanover St. P.O. Box 448 Manchester, NH 02105	Phone: (603) 518-4142
Gateway Community Services	2671 Huffman Blvd. Jacksonville, FL 32246	Phone: (904) 234-7398
Johnson County Mental Health Center	301 North Monroe St. Olathe, KS 66061	Phone: (913) 782-0283
Institute for Health & Recovery	349 Broadway Cambridge, MA 02139	Phone: (617) 661-3991
The University of Tennessee	324 Henson Hall Knoxville, TN 37996	Phone: (865) 974-3988
The Village	3050 Biscayne Blvd., Suite 900 Miami, FL 33137	Phone: (786) 312-6291

9) Payment to Respondents

Respondents will not receive any payment.

10) Assurance of Confidentiality

SAMHSA will not receive any client-level data. Additionally, the information collected through the AAFT Implementation Survey will be returned through a password-protected data file and stored on a secure server where access is limited to the staff directly responsible for the collection.

11) Questions of a Sensitive Nature

There are no questions of a sensitive nature collected in the three surveys.

12) Estimates of the Annualized Hour Burden

AAFT Implementation Surveys will be conducted with Principal Investigators/Program Directors, Clinical Supervisors/Clinicians, and Evaluators/Data Managers at each grantee site. Staffing patterns at each grantee site vary greatly allowing us to estimate the total number of respondents for each category based on initial grantee proposals. Per site, we estimate that we will conduct surveys with 1-2 Principal Investigators/Program Directors (total=21), 4 Clinical Supervisors/Clinicians (total=56), and 2 Evaluators/Data Managers (total=28).

The annualized cost was determined by applying an estimated wage rate of typical respondents, using information contained in one grantee application. These wages represent value of time spent, not new payments or reimbursements to staff members. No direct costs will be incurred by respondents.

The amount of time that is estimated for completion by all grantees is 78.75 hours in each collection year. The total reporting burden for the three years is estimated at 236.25 hours. The burden estimates, summarized in the following tables, are based on the reported experience of grantees that completed the pilot AAFT Implementation Survey as well as those who participated as part of the Evaluation Advisory Workgroup.

Annual Reporting Burden—Summary Table

Data Collection Activity	Number of Respondents1	Responses per Respondent2	Total Responses	Average Hours per Response	Total Hour Burden	Wage Rate (hourly)	Total Hour Cost ($)
CY 2010-12 Annual Reporting Burden
AAFT Implementation Survey—Principal Investigator/Program Director	21	1	21	0.75	15.75	$50	$787.50
AAFT Implementation Survey—Clinical Supervisor/Clinician	56	1	56	0.75	42	$26	$1092.00
AAFT Implementation Survey—Evaluator/Data Manager	28	1	28	0.75	21	$15	$315.00
ANNUAL TOTAL:	105		105		78.75		$2194.50
Represents project staff at three distinct levels—administrators, clinical staff, evaluators—across 14 grantee sites. Number of respondents is an average of respondents per role based on staffing patterns described in grantee proposals. The AAFT Implementation Survey will be completed once by respondents at all 14 sites at the end of each project year.

• Grantee Data Submissions

Grantees will be responsible for completing and submitting the AAFT Implementation Survey at the end of each project year. A link to the AAFT Implementation Survey will be emailed to each respondent. Grantees will complete the web-based survey and responses will be compiled in the database for later analysis.

13) Estimates of Annualized Cost Burden to Respondents

There are no startup or capital costs, nor are there maintenance costs to the respondents.

14) Estimates of Annualized Cost to the Government

CSAT will coordinate, monitor, analyze and report the AAFT Implementation Survey data provided by the grantees to the Government Project Officer. The estimated budget for the AAFT Implementation Survey data collection activities is $14,825. Web-based services will be hosted for a contracted amount totaling $43,379. The federal employee expends 2% time overseeing the Program Evaluation for the AAFT Program, equaling $2,557. The estimated total cost to the government is $60,761.

15) Changes in Burden

This is a new data collection.

16) Time Schedule, Publication and Analysis Plans

16.a.Time Schedule

Tasks Dates

OMB Approval: Pending

Data Collection: One week after OMB approval.

Data Collection Ends: September 2012

Analysis of Data: ongoing--annually

16.b. Publication Plans

Data will be presented at annual Grantee meetings in order to provide a summary of performance overview of the entire group of attending Grantees. Information related to the development, implementation and outcomes of this initiative’s AAFT program will be disseminated through journal articles, monographs/fact sheets and national conferences.

16.c. Analysis Plans

The primary purpose of the AAFT Implementation Survey is to be able to gather longitudinal data (end of each of 3 project years) from a range of grantee personnel concerning their implementation of AAFT and describe the key contextual factors that may influence the implementation and success of these efforts, challenges and accomplishments of implementation efforts, including lessons learned and sustainability.

The survey measures a range of domains using mostly closed-ended questions, with some open-ended responses. While some data elements are from existing instruments (e.g., Organizational Readiness for Change, ORC), original question sets were also developed to capture other constructs such as reactions to Chestnut supports and perceived implementation of AAFT components. The areas to be analyzed will include:

• Grantee background/experience

• Degree of implementation of AAFT components; reactions to components

• Adaptations/modifications to the model

• Use/helpfulness of Chestnut/other implementation support services

• Readiness and perceived changes in clinical practice/behavior

• Perceived barriers encountered in implementation and compensatory strategies

• Use of outcome and other data

• Efforts to plan for sustainability.

Data from the web survey will play an integral role in the cross-site implementation evaluation of the AAFT 3 grantees.  The web survey is the only mechanism we have to assess the attitudes and opinions of staff across grantee sites.  Data from the web survey will be used in four main ways.  First, respondents’ scores on the various readiness attitudinal scales embedded in the survey will be calculated.  These values will be aggregated to the site (grantee) level and used as outcome measures that track at the site level the extent to which sites have embraced the concept of evidenced based practices and associated constructs.  Second, these measures will play the role of covariates in analyses examining concrete implementation measures.  For example, do sites that achieve higher levels of A-CRA/ACC implementation have staff that hold more positive attitudes towards family treatment?  Third, the survey data will enable us to examine correlational associations at the staff level between staff characteristics and various readiness factors.  For example, do staff with higher levels of education or more treatment experience have differing attitudes towards the AAFT model and its components?  Finally, the web survey will provide critical measures of staff reactions to the various implementation supports provided by Chestnut Health Systems.  These data will be valuable in their own right, for example in allowing us to feed back to stakeholders information on which supports are seen as helpful and which are not, and also valuable as covariates in analyses examining implementation.  For example, do sites where the staff perceive the CHS supports more negatively tend to use the supports less, and achieve lower levels of implementation? 

In all of these analyses we will primarily be using descriptive statistics such as ranges, means, medians, and frequency distributions taken at the site level and plotted graphically both across sites and over time.  Multivariate graphical analyses, where we summarize numerous variables at the site level in a condensed graphical format will help us identify patterns among sites.  If necessary, we may draw on more formal multivariate data reduction techniques such as cluster analysis, principal components analysis, or multidimensional scaling to help us examine and understand patterns among the many measures we are collecting.  Because the sample size at the site level is small (n=14), we do not anticipate any modeling of primary study outcome measures.  If, however, correlational analyses of web survey data at the staff level warrant it, we may construct linear models of key staff level variables, such as attitudes towards evidenced based practices, as a function of site and individual level factors such as age and experience.

17) Display of Expiration Date

The expiration date for OMB approval will be displayed.

18) Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submission.

B. Collection of Information Employing Statistical Methods.

1) Respondent Universe and Sampling Methods

There are fourteen programs funded under the CSAT Assertive Adolescent and Family Treatment Program. The respondent universe for the AAFT Implementation Survey will be Principal Investigators/Program Directors, Clinical Supervisors/Clinicians, and Evaluators/Data Managers at each grantee site. Staffing patterns at each grantee site vary greatly allowing us to estimate the total number of respondents for each category based on initial grantee proposals. Per site, we estimate that we will conduct surveys with 1-2 Principal Investigators/Program Directors (total=21), 4 Clinical Supervisors/Clinicians (total=56), and 2 Evaluators/Data Managers (total=28).

2) Information Collection Procedures

As described earlier, the web-based AAFT Implementation Surveys will be completed by Principal Investigators/Program Directors, Clinical Supervisors/Clinicians, and Evaluators/Data Managers at each grantee site.

The initial contact regarding survey participation at each grantee site will be made via email to Principal Investigators (PI) and Program Directors (PD). Introductory emails will be sent to PIs and PDs at each grantee site and will:

• Explain the purpose of the overall evaluation and AAFT Implementation Survey

• Request their site’s participation in the survey at all personnel levels (e.g., PI/PD, Clinicians/Clinical Supervisors, Evaluators/Data Managers)

• Alert them that they will be receiving an email to participate in a web-based survey.

Once the AAFT Implementation Survey has been introduced to administrators, similar introductory emails will be sent to clinical supervisors, clinicians, evaluators, and data managers at each site.

The second contact with grantee staff at all three levels will be an email with the survey link embedded in the body of the letter.

Once collected, data will be analyzed using SPSS.

3) Methods to Maximize Response Rates

The response rate for the AAFT Implementation Survey is expected to be 80 percent. Respondents will receive up to four follow-up reminders in advance of the deadline. Reminders will be delivered primarily via e-mail; however, we will conduct personal phone calls to those who do not respond to e-mailed reminders. Respondents will have the opportunity to stop and re-start the survey as needed until completion. All correspondence will be as personalized as possible—not appearing as mass mailing to all grantees (e.g., tailoring each e-mail to include agency name).

To maximize the response rate to the web-based survey, within three weeks of the deadline, a third e-mail will be sent out thanking those who have participated and reminding those who have not, to complete the survey. Again, a link to the survey will be embedded in the body of the letter to ensure easy access to the survey.

Response rates will be monitored and another reminder e-mail will be sent out to PIs and PDs noting the number of responses received from their grantee program. Included in the letter will be a request to administrators that they support their staff members’ participation in the survey. This e-mail with the number of responses from their site will be follow-up by individualized e-mails with the link to the appropriate survey to PIs/PDs, Clinical Supervisors/Clinicians, and Evaluators/Data Managers.

Within two weeks of the deadline, our next contact will be individual calls to PIs and/or PDs at grantee sites with non-respondents.

Our final set of contacts will include:

1. A “Final Reminder” e-mail, including a link to the survey, will announce the close of the survey (within 7 days of the e-mail being sent out); and

2. A “Final Day Reminder” e-mail, with survey link included, reminder grantees that the survey will be closed to them by the end of the following day.

4) Tests of Procedures

The AAFT Implementation Survey was piloted in March 2010 with respondents from several grantee sites. Most respondents participated as members of the Evaluation Advisory Workgroup while another site—Orlando, FL—was approached to participate based on their previous experience as an AAFT grantee. A total of nine grantee personnel representing six grantee sites participated in the pilot. Respondents had the opportunity to review and comment on the AAFT Implementation Survey format and questions.

The average amount of time to complete was estimated based on the surveys completed in the pilot process. While time to complete ranged from 24 minutes to 1 hour, the average amount of time to complete was estimated to be 45 minutes. It should be noted that the time to complete also included items related to feedback on the survey itself.

Overall, those who completed the surveys appeared to be able to follow all skip patterns and answer questions appropriate to their role/s in the grant. The following are suggestions that were pertinent to the survey itself or expressed by multiple respondents and the related changes that were incorporated:

• Some questions are not applicable to the respondent’s role on the grant: Clinical Supervisors and Clinicians suggested that we remove questions considered irrelevant to their role on the grant (e.g., questions regarding evaluation/research or administrative matters). A few respondents—independent evaluators—also felt that questions regarding staff turnover and the effects of turnover at the site were not applicable to their role on the project.

  • Specific items that did not directly pertain to a respondent’s role were removed from their version of the survey, where possible (e.g., removed questions regarding staff turnover from the Evaluator/Data Manager survey version). In other instances where we were interested in varying points of view, we kept items in the survey to ensure we obtain data from each level of grantee personnel.

• Unable to respond without consulting other grantee staff: Some Clinical Supervisors, Clinicians and Evaluators felt unable to answer some items in the survey without conferring with other grantee staff. For example, when asked to rate the most challenging barriers for their site to implementing the AAFT program, some staff members felt that they would need input from either clinical staff and/or evaluators/data managers before answering.

  • To clarify this issue for respondents, we added the following statement to the introductory page of the survey—“We are gathering data from many sources and believe it is important to collect information from many perspectives as well. As you complete this survey, PLEASE RESPOND TO THE QUESTIONS FROM YOUR OWN PERSPECTIVE—choosing an answer that best describes your experience or opinion.” We also changed the language introducing sections of the survey where respondents felt they couldn’t provide answers from their perspective (e.g., “based on your experience”).

• Unsure for whom/what they are responding: In several places throughout the survey, respondents stated that the point of reference (e.g., the individual completing the survey, the program, the organization) for questions about processes, practices, or other changes that have occurred as a result of the AAFT project were too vague. For example, the question, “To what extent have each of these components become part of the normal, day-to-day routine?”—received feedback asking what “the” was referring to—the routine of the program or clinician.

  • In areas where the point of reference was unclear to respondents, we added the words “program,” “your,” and “organization” to further clarify to whom/what the statements/questions were referring.

• Survey length: Survey length was noted as a problem by some respondents. Those who provided this feedback felt the survey overall was too long and/or complicated.

  • As noted above, the survey was considered “too long” by some respondents. Again, respondents’ time to complete ranged from 24 minutes to 1 hour, including providing feedback on the survey itself. While we recognize that the survey may be somewhat longer that desired, we believe we’ve used the minimum number of items to capture valuable information about the implementation process from the grantee perspective. The information contained in the survey is not available via any other data currently being collected. Additionally, several sections of the survey contain scales from already-existing instruments (e.g., Organizational Readiness for Change—ORC) and the scales were retained for reliability and validity. Some items were eliminated from the Clinical Supervisor/Clinician and Evaluator/Data Manager survey versions in response to feedback that certain questions did not pertain to their role—thus shortening the surveys’ length.

5) Statistical Consultants

Contractors/Statistical Consultants:

Name/Title	Address	Contact Information
Terri Tobin, Ph.D. Evaluation Director	Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776	Phone: (978) 261-1418 [email protected]
Nick Huntington, M.A. Senior Research Associate	Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776	Phone: (978) 261-1453 [email protected]
Ken Winters, Ph.D.	University of Minnesota Dept. of Psychiatry F282/2A West 2450 Riverside Avenue Minneapolis, MN 55454	Phone: (612) 273-9800 [email protected]
Charles Turner, Ph.D.	Oregon Research Institute 1715 Franklin Blvd. Eugene, OR 97403-1983	Phone: 541-484-2123 [email protected]

Federal Project Officers/Statistical Consultants

Name/Title	Address	Contact Information
Karl D. Maxwell, Task Order Officer	Center for Substance Abuse Treatment, SAMHSA 1 Choke Cherry Road Room 5-1062 Rockville, MD 20857	Phone: (240) 276-2824 [email protected]

List of Attachments:

A. AAFT Implementation Survey

1. Principal Investigators/Project Directors

2. Clinical Supervisors/Clinicians

3. Evaluators/Data Managers

B. Sample Introductory and Reminder emails

1. Principal Investigators/Project Directors

• Introductory email—to PIs.PDs

• Email invitation—PI.PD

• 2^nd reminder with tally—to PIs.PDs

• Final reminder—PI.PD

2. All other staff (clinical, evaluators/data managers)

• Email invitation—all other staff

• Final reminder—all other staff

3. All staff (general letters used regardless of project role)

• 1st reminder email—all staff

• Last day reminder—all staff

6