Sat-ed OMB Ss-b 8-7-14

STATE ADOLESCENT TREATMENT ENHANCEMENT AND DISSEMINATION (SAT-ED) AND STATE YOUTH TREATMENT ENHANCEMENT AND DISSEMINATION (SYT-ED) PROGRAMS BIANNUAL PROGRESS REPORT ON INFRASTRUCTURE DEVELOPMENT MEASURES

SUPPORTING STATEMENT

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

To meet the objectives of this evaluation, SAMHSA will conduct data collection activities designed to measure the effectiveness of infrastructure development measures that have been designed for grantees of the Cooperative Agreements for State Adolescent Treatment Enhancement and Dissemination (SAT-ED) and State Youth Treatment Enhancement and Dissemination (SYT-ED) programs. This information will be used to inform SAMHSA about progress in infrastructure and services enhancement and expansion at the state and site levels. It will also help states/territories/tribes as they develop processes to improve management of their grant projects and better serve adolescents and transitional-aged youth and their families with substance use or co-occurring substance use and mental disorders. The data collection procedures will be to distribute the progress report via email for all data collection activities; respondents will be asked to respond to the progress report questions and return the completed progress report electronically via email or, if they prefer, to print, complete, and return the progress report via mail.

B.1. Respondent Universe and Sampling Methods

To evaluate the effectiveness of infrastructure development measures designed for grantees of the SAT-ED and SYT-ED programs, all 35 grantee administrators (i.e., Project Directors and/or Program Managers) across three cohorts where SAT-ED and SYT-ED grants are being implemented are eligible to be progress reported. In addition, up to two other project staff members (e.g., Evaluators, Clinical Supervisors, Site Coordinators) are expected to assist with gathering information to complete the progress report. Administrators and other project staff members from Cohort 1 grantees (13 Project Directors and/or Program Managers and 26 other project staff members) will be asked to participate in the progress report up to 6 times, administrators and other project staff members from Cohort 2 grantees (10 Project Directors and/or Program Managers and 20 other project staff members) will be asked to participate up to 10 times, and administrators and other project staff members from Cohort 3 grantees (12 Project Directors and/or Program Managers and 24 other project staff members) will be asked to participate in the progress report up to 10 times. Because participants will be asked to respond to the biannual infrastructure development measure via email (or via mail if preferred), there is an opportunity to census administrators from all 35 grantees with a minimal level of burden. Up to three follow-up emails will be sent, followed by a telephone call, to encourage participants who have not responded to complete the progress report. Based on previous experience conducting similar progress reports of grantee administrators and results from the pilot test of the biannual infrastructure development measure, the selected methods are expected to yield a response rate of 100 percent.

B.2. Information Collection Procedures

To support this effort, a biannual infrastructure development measure (see Attachment B2) will be distributed via email to each grantee administrator (i.e., Project Directors and/or Program Managers) twice a year, at approximately 6-month intervals, over a period of up to 3 years (Cohort 1 grantees) or up to 5 years (Cohort 2 and Cohort 3 grantees); the biannual infrastructure development measure will be collected via email or mail (based on the recipients’ preference). Before the progress report is distributed, each individual will be provided a unique identifying number. As discussed in Section A.10, these numbers will be used to track progress report responses and to determine if additional followup is needed to achieve the desired response rate and will not be used to identify individual respondents.

Data collection will begin by sending the biannual infrastructure development measure to each grantee administrator via email. An email cover letter will inform respondents about the purpose and significance of the progress report questions, ensure confidentiality to the respondent, and encourage a response as soon as possible. Participants will be asked to respond to the progress report questions and return the completed progress report electronically via email or, if they prefer, to print, complete, and return the progress report via mail.

Within 2 weeks of progress report distribution, all respondents will be sent a reminder email thanking those who have already responded and asking those who have not responded to complete the progress report and return it via email or mail as soon as possible. Additional follow-up emails will be sent weekly for up to 3 weeks to individuals who have not responded to encourage them to complete the progress report. To ensure high response rates, all potential respondents will be called if they have not responded to the final email reminder.

B.3. Methods for Maximizing the Response Rate

To maximize initial response rates, the following protocols will be used to reduce the burden on grantee administrators. The biannual infrastructure development measure will be distributed to respondents via email on a semi-annual basis. Respondents will be asked to complete the progress report and submit their responses electronically or, if they prefer, to print, complete, and return the progress report via mail. This approach will reduce burden to respondents who choose to use email by eliminating the time it takes to write out responses on a paper-and-pencil progress report and the time associated with mailing a hard copy of the progress report back to the contractor.

Within 2 weeks of progress report distribution, reminder emails will be sent to encourage respondents to complete the progress report and return their completed responses via email or mail. Up to 3 additional follow-up emails will be sent weekly to anyrespondents who have not yet participated, followed by a telephone call 1 week after the final reminder email, to ensure high response rates. In addition, all respondents will be informed of the significance of the progress report to encourage their participation. Finally, the efficiency of the progress report and the assurance of confidentiality will make progress report completion more amenable to participants.

B.4. Tests of Procedures or Methods

Pilot tests of the biannual infrastructure development measure instrument were conducted with a subsample of the grantee population. An electronic version of the biannual infrastructure development measure was distributed via email and pilot tested with eight grantees. The number of respondents contributing information to each biannual infrastructure development measure (per grantee) ranged from two to seven respondents. The results indicated that it takes approximately 12.0 hours for the Project Director/Project Manager to complete the progress report and 7.2 hours for other staff members providing supporting information; for all respondents to complete the biannual infrastructure development measure, including the Project Director, the average time to complete the instrument was 19.2 hours.

Respondents were then asked to comment on the clarity of the questions and identify any problems or issues (i.e., contents and format of the biannual infrastructure development measure). Participants were asked to provide feedback on the appropriateness of the questions for the intended purpose. Attachment B1 provides a summary of pilot test feedback for the biannual infrastructure development measure instrument tested and outlines the changes that were made to the data collection instrument based on this feedback.

Several sections of the progress report (see Attachment B2) were completed by eight Cohort 1 grantees of the Cooperative Agreements for State Adolescent Treatment Enhancement and Dissemination (SAT-ED) program. The averaged time it took the all respondents of a grantee to complete the bi-annual progress report was about 19.2 hours; the average time it took the PD/PM to complete the bi-annual progress report was about 12.0 hours.

Issue

Changes Made

1. Time to complete bi-annual progress report by numerous respondents of each grantee (19.2 hours). 2. Assure that the language is inclusive of transitional-aged youth. 3. Amend questions around effectiveness to allow for quantification (to be able to easily and clearly share results). 4. No questions related to the number of adolescents in need of treatment for substance use disorders/ co-occurring mental health disorders for each grantee. 5. No instructions included.

1. Change was made to methods to primarily have Project Directors/Program Managers answer the bi-annual progress report (rather than multiple staff members at each grantee), with support from up to two staff members, rather than multiple staff members. 2. Minor edits to language to provide clarification. 3. Minor edits to questions 14, 20, and 21 to allow quantification of answers (this will also decrease participant burden time by providing optional ranges for these questions, rather than having them be open-ended). 4. Added two questions that capture this information. 5. Added instructions.

Comments: There were no complaints or other critical comments expressed by the eight respondents on the appropriateness of the questions or other topics.

B.5. Statistical Consultants

Agency Responsibility

Within the agency, the following individual will have oversight responsibility for all contract activities, including the data analysis:

Darren Fulmore, Ph.D.

Public Health Advisor

Center for Substance Abuse Treatment

Substance Abuse and Mental Health Services Administration

1 Choke Cherry Road, Suite 5-1099

Rockville, MD 20857

(240) 276-1567 phone | (240) 276-2960 fax

[email protected]

The representative of the contractor responsible for overseeing the planned data collection and analysis is to be determined, as SAMHSA will be soliciting bids for a contractor.

REFERENCES:

Office of National Drug Control Policy (2013). 2013 National Drug Control Strategy. Retrieved December 26, 2013 from http://www.whitehouse.gov/ondcp/2013-national-drug-control-strategy

Bureau of Labor Statistics, U.S. Department of Labor (2012). Social and Community Service Managers. In Occupational Outlook Handbook­, 2012–13 Edition. Retrieved November 19, 2013 from http://www.bls.gov/ooh/management/social-and-community-service-managers.htm

U.S. Department of Health and Human Services, Office of Disease Prevention and Health

Promotion. (2013). Substance Abuse. Retrieved December 26, 2013 from http://healthypeople.gov/2020/topicsobjectives2020/overview.aspx?topicid=40

List of Attachment

1. Bi-Annual Progress Report

3