OMB_Supporting Statement_B_Youth Programs_6-23-16

The NATIONAL Evaluation OF the Substance Abuse and

Mental Health Services administration’s Youth Programs

Supporting Statement

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The respondent universe for the interviews, provider survey and focus groups comprises representatives from the SYT grantee programs, representatives of each programs stakeholders and representatives of all pilot site provider organizations that are part of the SYT grantee programs. The sampling frame is a census of the provider and stakeholder organization identified by the grantee and through document review and environmental scan activities. The expected number of respondents for the interviews and survey are discussed in Section A.12.

2. Information Collection Procedures

Interview and Focus Group Guides

Project directors from each SYT grantee will initially be contacted via email regarding the upcoming sustainability and implementation interviews. Project director information already provided to SAMHSA and the evaluation team. For interviews with stakeholders and providers, grantee project directors will be asked to confirm lists of key stakeholders and providers already identified in project applications and reports.

The initial email (see Attachment 7) will provide a brief summary of the interview questions to help grantees, stakeholders, and providers identify which staff will be best able to the address the Implementation, Sustainability, and Stakeholder Interview Guide topics. The initial email will also provide a suggested 2 week window in which to complete the interview(s). The evaluation team will work with the interviewee to schedule the interview(s) through follow-up emails and telephone calls.

For telephone interviews and virtual focus groups, once an interview or focus group date is set, the interviewer will provide a 1-800 conference line number to the grantee staff, stakeholder, or provider staff participating in the interview/focus group. The interviewer will also send a short reminder a few days before the scheduled interview. Interviews and focus groups will include grantee staff respondents, a lead interviewer and a designated note taker. Each interview and focus group will also be audio recorded if the respondents provide permission. The audio recording will only be used for note checking and will be destroyed once notes are finalized by the interviewer and note taker.

A similar process will be followed for interviews and focus groups completed in person. The initial email (see Attachment 7) will provide a brief summary of the interview questions to help grantees, stakeholders, and providers identify which staff will be best able to the address the Implementation, Sustainability, and Stakeholder Interview Guide topics. The initial email will also provide a suggested window in which to complete the interview(s) in person. The evaluation team will work with the interviewee to schedule the in-person interview(s) through follow-up emails and telephone calls.

Provider survey

Similar to the Implementation, Sustainability, and Stakeholder Interview Guides, the Provider Survey will use a list of grantee providers confirmed by the grantee project director. A key informant at each provider organization, typically a director or manager within the organization most knowledgeable on the SYT-ED or SYT-I program, will be identified by the evaluation team and sent the initial provider survey email. The initial email will include all information to securely access the Provider Survey web instrument and provide a 2-week period during which the survey should be completed.

Brief reminder emails will be sent to providers to help ensure surveys are completed. If a provider requests a paper copy of the survey, it will be mailed securely via Fed Ex with a postage-paid return envelope. Full details and template emails are provided in Attachment 8.

Respondents to the Provider Survey will be provided with a user name and password (changed by the respondent at first login) through which to access the survey. Before the respondent begins the survey, the respondent will be asked to review a short paragraph that details how the Provider Survey is private and voluntary. Surveys will only be considered complete once the respondent submits the survey; this feature will allow the respondent to complete parts of the survey and return to the survey at a later time. All data will be stored securely on RTI’s servers and data will only be identified with a random provider ID.

3. Methods to Maximize Response Rates

While grantees in the SYT programs are expected to participate in evaluation data collection activities as SAMHSA grantees, the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, focus groups are designed to help ensure that each grantee, stakeholder, and provider responds. Efforts to reach a 100% response rate begins at recruitment and focuses on reducing grantee burden and communicating information about the data collection efforts before its implementation. The evaluation team has already begun the engagement process by participating in grantee meetings and conference calls.

The evaluation team plans to use reminder emails and telephone calls to help maintain a 100% response rate.

4. Test of Procedures

Both the Implementation, Sustainability, and Stakeholder Interview Guides, Provider Survey, and focus groups leverage the significant experience of the evaluation team and both instruments are based on tools used successfully in other evaluation efforts. The Interview and Focus Group Guides are adapted to address the specific youth program evaluations from similar instruments that were used successfully in other evaluations including the SAMHSA Access to Recovery evaluation (used with 25 grantee projects and over 200 respondents), ASPE Medicaid Expansion Evaluation, and the SAMHSA Homeless Programs Evaluations (used with 43 grantee projects and over 400 respondents). The Provider Survey is adapted from similar provider survey used in the Access to Recovery evaluation which received hundreds of responses.

The evaluation team also tested instruments by completing instruments to estimate the burden time as outlined in Section A.12.

5. Statistical Consultants

As noted in Section A.8, SAMHSA has contracted an evaluation team consisting of relevant subject-matter experts. In addition, the contractor team comprises several experts who will be directly involved in the data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, economic analysis, database management, and data analysis. Exhibit 11 provides details of these team members and advisors.

Exhibit 11. Data Collection and Analysis Team Members and Advisors

Expert	Affiliation	Contact Information
Sharon Amatetti, MPH Senior Public Health Analyst SAMHSA Women’s Issue Coordinator	SAMHSA/CSAT	5600 Fishers Lane Rockville, MD 20857 [email protected]
Twyla Adams Branch Chief	SAMHSA/CSAT	5600 Fishers Lane Rockville, MD 20857 [email protected]
Linda White-Young Public Health Advisor	SAMHSA/CSAT	5600 Fishers Lane Rockville, MD 20857 [email protected]
Arnie Aldridge, PhD Research Economist	RTI, International	3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 [email protected]
Josephine D. Korchmaros, PhD Director—Research Methods and Statistics	University of Arizona-Southwest Institute for Research on Women	181 S. Tucson Blvd., Ste. 101 Tucson, AZ 85716 [email protected]
Will Parish, PhD Research Economist	RTI, International	3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 [email protected]
Michael Penne Research Statistician	RTI, International	3040 East Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 [email protected]

ATTACHMENTS

Attachment 1 – Evaluation Questions

Attachment 2 – Implementation Interview Guides

Attachment 3 – Sustainability Interview Guides

Attachment 4 – Stakeholder Interview Guides

Attachment 5 – Provider Survey

Attachment 6 – Focus Group Guide

Attachment 7 – Interview Guide Supporting Documents

Attachment 8 – Provider Survey Supporting Documents

Attachment 9 – Provider Survey Table Shells

3