OMB_Supporting Statement B_NITT-HT_1-28-16

NOW IS THE TIME (NITT) – HEALTHY TRANSITIONS (HT) EVALUATION

SUPPORTING STATEMENT

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

The NITT-HT evaluation will use a combination of a census approach and a nominating approach to collecting process and outcomes data through the instruments at the center of this OMB application, while using existing DCI grantee and client level outcomes data and data from survey samples for the SYAI. The procedures and source for collecting data are listed in Table 9 below.

Using a census approach, the targeted universe is all NITT-HT grantee projects (n=17) and learning laboratories (n=43). All grantees and learning laboratories have all agreed to participate in NITT-HT evaluation data collection activities as a condition of funding.

Project Staff. All 17 grantee project directors are expected to complete a web-based Project Director Web Survey, Project Director Telephone Interview and a Services & Supports Inventory administered via a telephone interview.

Core staff, including the youth coordinator and peer staff, at all learning laboratories are expected to complete the Core Staff Web Survey and the Youth Coordinator In-Person Interview during the site visit. Core staff include those knowledgeable about the process of developing and delivering services for youth/young adults at the learning laboratory.

A census of all 17 NITT-HT grantee project directors and learning laboratories is necessary due to the heterogeneous nature of the NITT-HT programs. These programs encompass a wide variety of organizational types and structures that are implementing a range of interventions with various outcome goals. The variety between the programs makes it critical to the evaluation to capture the details of each program to be able to answer the evaluation questions and assess which program characteristics and mix of interventions are associated with better outcomes for this youth/young adult demographic group and types of communities. Additionally, these data will be used by SAMHSA to monitor each program’s performance and grantee and learning laboratories will also use it to track their ongoing implementation.

Other Providers and Stakeholders. A sampling approach for the Provider In-Person Interview and the Other Stakeholder In-Person Interview, conducted during the grantee visit, includes nomination by the grantee project directors to identify key informant (project directors can self-nominate for this task if they believe they would be the most appropriate key informant for the task, or can nominate others such as the grantee local evaluator).

Community and State Leadership. Using a nominating sampling approach, grantee project directors will be asked to nominate community, state and grantee leadership to conduct the CSTI and SSTI. The Collaborative Member Survey will be administered to a subset of CSTI respondents who participate in a NITT-HT grantee’s Advisory Team and the Collaborative Self-Assessment will be completed by one representative of each NITT-HT Advisory Team within the NITT-HT grantee community.

The CSTI is designed to be administered to stakeholders at the community level who have knowledge about system-level collaboration and systems change. CSTI was designed to serve as a guide to help implementing communities understand both what they are aiming for—sustainable capacity—and how much progress they have made in achieving that goal. CSTI is completed by stakeholders in the local community who are knowledgeable about the process of developing and delivering services for young adults. This usually includes the local advisory or steering committee, staff of the NITT-HT program, staff of agencies providing portions of the services, and young adult and family members’ advocates. Young people, family members, and other allied adults who serve on advisory boards or who are significantly involved with planning and evaluating services for emerging adults are also asked to respond to the CSTI.

The SSTI is designed to be administered to individuals at the state level who are involved with planning and funding services for emerging adults with mental health needs. This list usually includes administrators or staff from state agencies for mental health, child welfare, education, and vocational rehabilitation, as well as other state agencies responsible for some aspects of services to young people. SSTI is also completed by youth/young adult and adult allies who are active in promoting, planning, or overseeing services at the state level, as well as other members of state-level advisory groups or governing bodies.

Youth/Young Adults & Family/Adult Allies. The Supplemental Youth and Young Adult Interview (SYAI) will be conducted with transition-age youth/young adult participants at program entry, 12-months, and 24–month follow-up. This supplemental participant-level data will be in addition to the client-level DCI data that grantees submit. These supplemental youth/young adult interviews will provide participant-level information not currently required by SAMHSA but necessary to fully achieve the objectives of the NITT evaluation. These interviews will be administered per voluntary consent to participants who receive services through NITT-HT initiative. The contractor conducted a statistical power analysis to determine an appropriate sample size. For a design with 17 grantees, conducting 90 interviews per grantee would have at least 80% power to detect effect sizes of d greater than |0.20| so long as the grantee-level intraclass correlation does not exceed 0.14. The target cases will be evenly allocated across the number of participating learning laboratories within each grantee program. Supplemental youth/young adult interviews will not begin until a learning laboratory has been actively serving clients for at least 6 months and served at least 15 youth/young adults. This sampling plan is designed to reduce burden and sample target cases for participation in the supplemental interviews only after learning laboratories have completed program development and are through the early stages of program implementation.

Young Adult and Family/Adult Ally Focus Group will be conducted with youth/young adults directly involved in NITT-HT system change efforts, youth/youth adults who are recipients of NITT-HT services, family/adult allies of youth/young adults service recipients, and family/adult allies involved in NITT-HT grantee planning and systems change efforts. The focus groups will be conducted during the grantee visit and the sampling approach will include participant nomination by grantee staff. Participation in the focus group is on a voluntary basis.

B.2. Information Collection Procedures

NITT-HT cross-program evaluation data collection will be centralized and coordinated through the NITT-HT data collection team leader and liaison assigned to each grantee learning laboratory. This liaison will be the grantee learning laboratory’s primary point of contact for all questions about data collection, reporting, and feedback. Furthermore, these liaisons will help collect data. For example, all process evaluation telephone interviews will be conducted by the UMMS or RTI liaison assigned to that individual grantee and learning laboratory.

The NITT-HT evaluation approach will use various data collection strategies across the range of respondents, including perspectives of stakeholders, youth/young adults as well as the system (i.e., focusing on learning laboratories and grantees). Due to the large number of relevant stakeholders at multiple levels of analysis, many types of instruments and modalities are being used; however, the evaluation has been designed to minimize burden for each stakeholder type. The type of data collection, respondent/source, mode, administration frequency and expected sample size are summarized in Table 9. This table is organized by key information sources:

• community and state leadership;

• NITT-HT project staff;

• other providers and stakeholders;

• youth/young adults and family/adult allies

• grantee documents and data; and

Table 8. NITT-HT Data Collection Procedures by Source of Information

Type of Data Collection	Respondent/ Source	Mode	Administration Frequency	Expected N (per administration)
COMMUNITY AND STATE LEADERSHIP
Community Support for Transition Inventory (CSTI)	Community leadership	Web-based survey	Twice – Years 2, 4	15–25 per learning laboratory (maximum total n = 1075)
State Support for Transition Inventory (SSTI)	State/Grantee Leadership	Web-based survey	Twice – Years 2, 4	20–25 per grantee (maximum total n = 425)
Collaborative Member Survey	Respondents to the CSTI who participate in Advisory Teams	Web-based survey	Twice-Years 3, 5	15–25 per learning laboratory (maximum total n = 1075)
Collaborative Self-Assessment	One representative of each Advisory Team	Web-based survey	Twice-Years 3, 5	1 per learning laboratory (maximum total n = 43)
PROJECT STAFF
Project Director Web Survey	Project Director	Web-based survey	3x (prior to each PD Telephone Interview)	1 per grantee (17 total)
Project Director Telephone Interview	Project Director	Phone interview	3x (twice prior, once following grantee visit)	1 per grantee (17 total)
Services & Supports Inventory	Local Evaluator, Project Director, or Other Knowledgeable Key Informant	Phone interview	Once (prior to grantee visit)	1 per grantee (17 total)
Core Staff Web Survey	Learning laboratory “core staff”	Web-based	Once (prior to grantee visit)	All core staff, max of 10 per implementing learning laboratory

(continued)

Table 8. NITT-HT Data Collection Procedures by Source of Information (continued)

Type of Data Collection	Respondent/ Source	Mode	Administration Frequency	Expected N (per administration)
Youth Coordinator In-Person Interview	Learning laboratory youth coordinators and peer staff	In-person interview during grantee visit	Once	Youth coordinators and up to 2 peer staff per learning laboratory
OTHER PROVIDERS AND STAKEHOLDERS
Provider In-Person Interviews	Key informants for contracted providers nominated in Services & Supports Inventory	In-person interview during grantee visit	Once	3–5 per grantee (51–85 total)
Other Stakeholder In-Person Interviews	Community stakeholders (i.e., non-staff) nominated in Services & Supports Inventory	In-person Interview during grantee visit	Once (prior to grantee visit)	3 per grantee (51 total)
YOUTH/YOUNG ADULTS & FAMILY/ADULT ALLIES
Supplemental Youth & Young Adult Interview	Participants who receive services through NITT-HT initiative	Web-based (baseline); Web or telephone (12- and 24-months)	3x (baseline, 12- and 24-months)	90 per grantee (1,530 total)
Young Adult Focus Group Guide & Information Form	Young adults nominated by learning laboratories	Focus group & paper-and-pencil Information Form	Once	2 groups per implementing learning laboratory
Family/Adult Ally Focus Group Guide & Information Form	Families & adult allies nominated by learning laboratories and young adults	Focus group & paper-and-pencil Information Form	Once	2 groups per implementing learning laboratory
GRANTEE DOCUMENTS AND DATA
Document Review	Individual young adult service records	Document review	Once (during grantee visit)	5–10 per implementing learning laboratory

Web-based Instruments

Each web-based data collection instrument (with the exception of the SYAI, see below for more detail on the SYAI) will follow the procedures outlined below.

Emails will be sent to the designated respondents to inform them that data collection has started. A link to enter the web-based survey will be included in the email, as well as information on the background, purpose, types of questions and length of the survey. When respondents complete the survey a “thank you” email will be sent automatically. Nonresponders will be sent predefined, automated weekly emails as needed, to remind them to complete their survey. If the survey is not completed within 3-weeks, the grantee or learning laboratory will be contacted via telephone by the assigned liaison.

The NITT-HT evaluation team will develop user manuals for accessing and navigating the online data collection systems and question-by-question and frequently asked question (FAQ) guides to help respondents accurately complete the surveys. Grantees and learning laboratories will also be provided training webinars to: 1) walk through the online data collection systems, and 2) review the instrument data collection procedures and answer questions. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

Availability is important in any data collection system, especially one employed by grantees and learning laboratories around the country, including multiple time zones and pacific jurisdictions. The online system will be maintained in an available state as much as possible to allow grantees and learning laboratories access for viewing their specific data, as well as to give the NITT-HT team, grantees, and SAMHSA access to reports.

Providing a robust system that is simple and easy to use across all areas is also critically important. To achieve this, the contractor will implement user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees. Additionally, every page of the online data system will have a “Help” or “Support” link clearly labeled, which will allow the respondent to access the following support resources:

1. Search. More comprehensive than a list of FAQs and more organized than a support forum, the search feature will offer a “layered information” approach so that respondents can search by keyword and then drill down to view material at increasing levels of detail. It will be a curated and easily searchable source of information including items such as

• user guides,

• data collection protocols,

• training materials and webinar,

• Question-by-Question (QxQs), and

• FAQs.

1. Contact Us. Respondents may request assistance either by calling a provided toll-free number or sending an email request, as desired. The toll-free line will be routed to an email system that is checked regularly by members of the training and technical assistance team. Staff responding to technical assistance requests will be trained in use of the system and have ready access to the full support system. Training and technical assistance team staff will monitor all submitted tickets to ensure timely response and resolution of technical assistance requests.

Grantee Telephone Interviews

Respondents to the various telephone interviews will be contacted by the NITT-NT evaluation via email with telephone follow-up to setup a mutually convenient time for the interview during regularly scheduled business hours. Before conducting the telephone interviews, the evaluation team will review grant applications (submitted to SAMHSA by each grantee and given to the evaluation team by SAMHSA) and other documents (e.g., grantee Year 1 planning documents, local evaluation plans, progress reports) that detail the characteristics of the program and abstract information relevant to the evaluation (e.g., project structure, interventions) so that liaison staff conducting the interviewers are familiar with the grantee. This preabstracted information will be used to prepopulate some interview questions to reduce respondent burden. For instance, a list of the grantee’s learning laboratories and client base will be prepopulated and confirmed or updated with the respondent, as opposed to asking the respondent to generate the list while on the telephone.

Once the interview is scheduled, the contractor will provide the participant with an electronic version of the consent form and the partially prepopulated interview guide and a toll-free, passcode-protected telephone conference number. Before beginning the telephone interview, consent will be requested to record the interview to confirm, if needed, the accuracy of noted responses. A senior evaluator from the contractor’s evaluation team will lead the respondent through the interview while a junior evaluator will record responses and take notes. After the interview, the interviewer and note taker will review the responses for accuracy. Any areas of discrepancy will be validated with the recording (if consented by the respondent); once the responses are considered final, the recording will be deleted. An electronic version of the telephone interview will be maintained on a password protected, secure server accessible only to the contractor’s evaluation team. After the interview, the interviewer will send an email thanking the respondent for his or her participation.

This procedure will be followed for the follow-up data collection instruments as well.

A procedures manual will be developed for the administration of the telephone interviews and training will be provided to all interviewers and notetakers to walk through interview procedures and questions.

Supplemental Youth & Young Adult Interview (SYAI)

The SYAI will be self-administered by the participant at three time periods – baseline, and 12- and 24-month follow-up. All three interviews will be conducted via web-based programs with audio capabilities on tablet computers and headphones provided to sites by the evaluation team. The self-administered web-based administration with audio capabilities is designed to protect client privacy, reduce discomfort and burden, and ensure that the data are of the highest quality. The SYAI will not to begin until a learning laboratory has been actively serving clients for at least 6 months and served at least 15 youth/young adults.

The supplemental interview will be administered at baseline by grantees in conjunction with the baseline DCI interviews to youth/young adults ages 16–25 years. The 12- and 24-month follow-up interviews will also be made available to complete on tablets at the grantee sites. Participants can complete the follow-up web-based survey on a personal computer, via telephone with an interviewer trained to conduct interviews with youth/young adults, or on the designated NITT-HT tablet at the learning laboratory.

To begin the SYAI baseline, the learning laboratory selected staff will provide the client with a brief introduction to the interview and ask the client if they will agree to hear more. If the client agrees to proceed, the staff will read the informed consent for the client interview aloud to the client, answer any questions and ask if the client understands and agree to its contents, the staff will instruct the participant client to click the consent button on the consent page of the web-based program to acknowledge review and submit consent. A participant cannot complete the web-based instrument until they have submitted consent. The participant will then be given the consent information page for their records. Participants will also be required to submit consent at both the 12- and 24-month follow-up interviews, whether conducted via web-based or telephone interviews.

To increase response rates, all clients who agree to participate in the SYAI at baseline will receive a gift card respondent payment worth a $20 value. Clients who agree to participate in the 12- and 24-month follow-up will receive a gift card respondent payment worth a $25 value.

Focus Groups

Prior to beginning a focus group discussion, the participants, including both youth/young adults and family/adult allies, will be read and provided a copy of the focus group consent form that informs participants of their rights, including the right to not answer any question or participate in any discussion, and asks for their written consent to participate in the discussion and for the discussion to be recorded. If a participant does not provide written consent to participate then that individual will be excused from the group and provided with the respondent payment in appreciation for their time. Consent will have to be provided by parents or guardians for any youth participants under the age of 18 years. Youth will then need to provide their assent for participation in the focus group. Participants will also be asked to complete the Focus Group Participant Information Form at the conclusion of the focus group. The form will include basic background information for analysis purposes only (e.g., demographics, extent of experience with the mental health system and grantee community). Recordings will be used to ensure that information is correctly captured from multiple focus groups, that information has been consistently captured and to correct and clarify brief written notes as needed and part of data quality assurance process. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed. Information collected by the site visit interviews is only reported in aggregate and individual respondents will not be identified.

Grantee Site Visit

Prior to beginning a site visit discussion, the respondent(s), including both project staff and key informant/stakeholder, will be read and provided a copy of the site visit consent form that informs participants of their rights, including the right to not answer any question, and asks for their written consent to participate in the discussion and for the discussion to be recorded. Recordings will be used to ensure that information is correctly captured from multiple interviews, that information has been consistently captured and to correct and clarify brief written notes as needed and part of data quality assurance process. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed. Information collected by the site visit interviews will be reported only in aggregate and individual respondents will not be identified.

B.3. Methods to Maximize Response Rates

Grantees are required to participate in all NITT-HT evaluation activities by the terms and conditions of the NITT-HT grant award. The NITT-HT evaluation team will employ a number of strategies to help ensure grantees and learning laboratories participate with a 100% response rate.

Web-based Surveys

As described above, the NITT-HT evaluation will develop user manuals for accessing and navigating the online data collection systems and question-by-question and FAQ guides to help respondents accurately complete the instruments. Grantees will also be provided training webinars to introduce the NITT-HT evaluation, walk through the online data collection systems, and review data collection procedures and instruments. Within the online data collection system, all manuals, guides, and training webinars will be archived and accessible to respondents for reference at any time.

For online web-based surveys, grantees and learning laboratories will be sent automated, predefined emails to remind them of completion deadlines. Specifically, the following reminder schedule will be followed:

1. Start of Data Collection: At the start of data collection when the web-based survey is available for the participant to complete, grantees and learning laboratory staff previously nominated will be sent automated emails to inform them that they were nominated by the project director to complete the survey and of the start of data collection. Emails will include a link to the survey as well as login and password the participant will need to access the survey.

2. At Survey Completion: The data submission system will automatically send a “Thank You” email when respondents complete the survey.

3. Weekly After the Start of Data Collection: The web-based system will automatically send an email to participants who have not completed the survey reminding them that data collection has started and asking them to complete the survey at their earliest convenience. The email will include the evaluation toll-free number and email address should the participant have any questions or need to speak with the evaluation team liaison.

4. One Week Before the Stated Data Collection Deadline: The evaluation team liaison will call the designated participant alerting them that the data collection deadline is approaching and offer any necessary assistance with completing the web-based survey.

5. Data Collection Deadline: The web-based system will automatically send an email to nonresponders and their grantee alerting them that the data collection deadline has passed. When a nonrespondents is a grantee, the SAMHSA Grantee Project Officers (GPOs) will also be notified.

6. Two Weeks After Stated Data Collection Deadline: NITT-HT will notify the GPO, who will either email or request a telephone call with grantees (or with learning laboratories and their respective grantees) who have not completed their survey two weeks after the stated deadline. Grantees will be expected to monitor their learning laboratories compliance.

For the following telephone or in-person interviews, the initial email invitations will provide a thorough explanation of the study and its importance, the reasons the participant is being asked to participate, and means by which they can contact the evaluation team for additional information, including a toll-free telephone number and project specific email.

Telephone Interviews

• Project Director Telephone Interview

• Services & Supports Inventory

In-Person Interviews During Site Visit

• Youth Coordinator In-Person Interview

• Other Stakeholder In-Person

• Provider In-Person Interview

• Young Adult Focus Group

• Family/Adult Ally Focus Group

The evaluation team will aim to identify the most convenient time for the participant to complete the telephone or in-person interview. Before the interview, participants will also be provided the interview topics so they will be knowledgeable about the types of information to be collected. Nonresponders to the initial email invitation will be sent weekly follow-up reminder emails. If needed—although the evaluation team does not anticipate that it will be necessary—the follow-up reminder emails will include the grantee’s GPO.

Supplemental Youth & Young Adult Interview (SYAI)

To maximize response rates and ease burden for the participating client, the SYAI will be completed at the same time as the CDP DCI, a data collection instrument that NITT-HT grantees are required to administer with each NITT-HT client, one completed immediately after the other. The process of implementing the SYAI maximizes response rates with a self-administered web-based SYAI at baseline, 12- and 24-month follow-up on a tablet computer with audio capabilities on headphones is designed to protect client privacy, reduce discomfort and burden, and ensure that the data are of the highest quality. Though the SYAI will be self-administered by the participating client, the contractor is asking the NITT-HT grantee learning laboratory staff to assist with administering the SYAI at baseline and again at the 12- and 24-month follow-up if the client prefers to complete the interview on the designated NITT-HT evaluation tablet. Initially it will be important to have someone familiar with the youth/young adult to explain the purpose of the NITT–HT evaluation to bolster participant engagement and maximize response rate in the supplemental interviews. The contractor will hold training sessions with all grantee learning laboratory selected staff to detail the steps involved in initiating the administration of the client interview and the procedures to follow to ensure protection of respondents’ rights and safeguarding of client data. Client interview informed consent and a frequently asked questions (FAQ) will be accessible via the survey web-based program and provided to the grantee learning laboratory staff as a handout to reference and give to clients for their records.

To maximize response rates by utilizing the contractor’s extensive experience conducting longitudinal data collection, the 12- and 24-month contacting will be conducted by the contractor. In preparation for the follow-up interviews, the contractor will email participants regarding the interview, including a link to the web-based interview to complete on a personal computer or the option to return to the learning laboratory and complete on the designated NITT-HT evaluation tablet. The email will also include an evaluation team contact to call toll-free or e-mail with questions. Participants who have not completed a follow-up interview after a week of the contractor sending the initial email invitation will receive an email reminder asking them to complete the interview at their earliest convenience. Clients will receive one email reminder per week for up to three total email reminders. The third email reminder will inform the client that they will receive a call from an NITT-HT interviewer to provide them the option of completing the interview on the phone. One -week after sending the third email reminder, or one month after sending the initial email invitation, nonresponders will start receiving calls from the contractor’s trained interviewer staff giving them the option to complete the interview on the phone.

NITT-HT clients are transition age youth/young adults with mental health disorders, a group that can be hard to reach. To further increase response rates, all clients who agree to participate in the SYAI at baseline will receive a gift card respondent payment worth a $20 value. Clients who agree to participate in the 12- and 24-month follow-up will receive a gift care respondent payment worth a $25 value. Respondents will not be penalized if they wish to skip questions or stop the interview before completing the baseline or either of the follow-up interviews.

The contractor will apply their extensive experience studying longitudinal outcomes among youth/young adults by engaging participants in social media and periodic check-in emails and/or texts (e.g., birthday, quarterly mental health awareness tips, all contacting of their choice as indicated in the SYAI) to keep participants engaged. The contractor will maintain social media pages to bolster follow-up rates and encourage youth/young adult participation.

B.4. Test of Procedures

Contractor and subcontractor staff completed the web-based surveys and interview instruments, either in paper-pencil form or within word processing software. These staff members have experience with evaluation initiatives, as well as calculating survey lengths. It is likely that the web-based versions of the surveys below will take less time than the paper version tested to generate the estimates in this section, as skip patterns will be automated and some items will be prepopulated automatically after initial responses.

The CSTI takes 24 minutes to complete, including time for reading the survey instructions and consent and responding to survey questions. This time represent actual averages taken across three randomly selected communities in which the survey was previously fielded. These tests of the CSTI were developed as part of another study that did not require OMB approval.

The SSTI takes 19 minutes to complete, including time for reading the survey instructions and consent and responding to survey questions. This time represent actual averages taken across three randomly selected communities in which the survey was previously fielded. These tests of the CSTI were developed as part of another study that did not require OMB approval.

The Collaborative Member Survey is estimated to take 15 minutes to complete per response.

The Collaborative Self-Assessment Survey is estimated to take 50 minutes to complete per response; this includes time to look up and compile information and time to complete the web-survey.

The Project Director Web Survey is estimated to take 20 minutes to complete.

The Project Director Telephone Interview is estimated to take 120 minutes to complete; this includes time to look up and compile information and time to complete the telephone interview.

The Services & Support Inventory is estimated to take 40 minutes to complete.

The Core Staff Web Survey is estimated to take 25 minutes to complete per response.

The Youth Coordinator In-Person Interview is estimated to take 60 minutes to complete.

The Provider In-Person Interview is estimated to take 45 minutes to complete per response.

The Young Adult Focus Group conducted during the grantee site visit will take 105 minutes (one hour and forty-five minutes) to complete; this includes time for introductions, directions and informed consent, time for the focus group discussion, and time to complete the paper-and-pencil Information Form. The focus group discussion will end after 105 minutes, regardless of whether all discussion items have been covered.

The Family/Adult Ally Focus Group conducted during the grantee site visit will take 105 minutes (one hour and forty-five minutes) to complete; this includes time for introductions, directions and informed consent, time for the focus group discussion, and time to complete the paper-and-pencil Information Form. The focus group discussion will end after 105 minutes, regardless of whether all discussion items have been covered.

The Document Review conducted during the grantee site visit is estimated to take learning laboratory staff 15 minutes; this includes time to look up and compile the files/charts and time to make files/charts available to the evaluation team members to conduct the on-site review.

The Stakeholder In-Person Interview is estimated to take 45 minutes to complete per response.

The SYAI was pilot tested with 4 young adults through the YAB. The SYAI is estimated to take 40 minutes, including time for reading and reviewing survey instructions and consent and responding to survey questions.

B.5. Statistical Consultants

As noted in Section A.8, SAMHSA has consulted with an expert panel on the NITT-HT evaluation plan, data collection procedures, and analysis plans. These experts will continue to provide advice and feedback throughout the course of the evaluation through annual panel meetings. In addition, the contractor team comprises several experts who have been involved in the development of the NITT-HT data collection and analysis plans and will be directly involved in data collection and statistical analysis. Also, SAMHSA advisors will be consulted throughout the evaluation on various statistical aspects of the design, methodological issues, and data analysis. Table 10 provides details of these team members and advisors.

Table 9. Statistical Consultants for the Now Is the Time (NITT) – Healthy Transitions Evaluation

Name & Role in Evaluation	Title & Address	Contact Information
NITT-HT Evaluation Staff
James Trudeau, Ph.D. NITT Evaluation Project Director	Senior Research Social Scientist Center for Justice, Safety, and Resilience RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-485–7751 Email: [email protected]
Heather Ringeisen, Ph.D. NITT-HT Evaluation Lead & NITT Evaluation Deputy Project Director	Director Center for Behavioral Health and Development RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-541-6931 Email: [email protected]
Antonio Morgan-Lopez, Ph.D. NITT-HT Analysis Team Lead	Principal Scientist Risk Behavior and Family Research RTI International 3040 East Cornwallis Road Research Triangle Park, NC 27709	Phone: 919-316–3436 Email: [email protected]

(continued)

Table 9. Statistical Consultants for the Now Is the Time (NITT) – Healthy Transitions Evaluation (continued)

Name & Role in Evaluation	Title & Address	Contact Information
Maryann Davis, Ph.D. NITT-HT Process Evaluation Co-Lead	Research Associate Professor of Psychology Transitions RTC, Systems and Psychosocial Advances Research Center Department of Psychiatry, University of Massachusetts Medical School 222 Maple Avenue, Chang Building Shrewsbury, MA 01545	Phone: 508-856-8718 Email: [email protected]
Mason Haber, Ph.D. NITT-HT Process Evaluation Co-Lead	Assistant Professor of Psychology Transitions RTC, Systems and Psychosocial Advances Research Center Department of Psychiatry, University of Massachusetts Medical School 222 Maple Avenue, Chang Building Shrewsbury, MA 01545	Phone: 508 856-8487 Email: [email protected]
Nancy Koroloff, Ph.D. NITT-HT Portland State University Co-Lead	Co-Director RTC, Pathways to Positive Futures Portland State University 1600 SW 4th Avenue, Suite 900  Portland, OR 97201	Phone: 503-725-9675 Email: [email protected]
Janet Walker, Ph.D. NITT-HT Portland State University Co-Lead	Co-Director RTC, Pathways to Positive Futures Portland State University 1600 SW 4th Avenue, Suite 900  Portland, OR 97201	Phone: 503-725-8236 Email: [email protected]
SAMHSA Advisors
Nainan Thomas, Ph.D. Contracting Officer’s Representative	Public Health Advisor CMHS, SAMHSA 1 Choke Cherry Road, Room 6-1099 Rockville, MD 20857	Phone: 240-276-1744 Email: [email protected]
Kirstin Painter, Ph.D. Alternate Contracting Officer’s Representative	Public Health Advisor CMHS, SAMHSA 1 Choke Cherry Road, Room 6-1040 Rockville, MD 20857	Phone: 240-276-1932 Email: [email protected]
Beda Jean-Francois, Ph.D.	Social Science Analyst CBHSQ, SAMHSA 1 Choke Cherry Road, Room 2-1012 Rockville, MD 20857	Phone: 240-276-0370 Email: [email protected]

REFERENCES

Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res, 6(2), 61-74.

Bandura, A. (1989a). Multidimensional Scales of Perceived Self-Efficacy. Psyctests. doi: 10.1037/t06802-000

Bandura, A. (1989b). Multidimensional scales of perceived self-efficacy. Unpublished test. Stanford, CA: Stanford University.

Brady, M. P., Rosenberg, H., & Frain, M. (2008). Job observation and behavior scale: Opportunity for self-determination, standardization and factor analysis. Career Development for Exceptional Individuals, 31, 175-185.

Brady, M. P., Rosenberg, H., & Frain, M. (n.d.). Job Observation and Behavior Scale: Opportunity for Self-Determination. Psyctests. doi: 10.1037/t06061-000

Callahan, S. T., & Cooper, W. O. (2004). Gender and uninsurance among young adults in the United States. Pediatrics, 113(2), 291-297.

Cheong, JeeWon, MacKinnon, David P., & Khoo, Siek Toon. (2003). Investigation of Mediational Processes Using Parallel Process Latent Growth Curve Modeling. Structural Equation Modeling: A Multidisciplinary Journal, 10(2), 238-262. doi: 10.1207/S15328007SEM1002_5

Cook, J. R., & Kilmer, R. P. (2010). The importance of context in fostering responsive community systems: supports for families in systems of care. American Journal of Orthopsychiatry, 80(1), 115-123. doi: 10.1111/j.1939-0025.2010.01014.x

Dennis, M. L., Chan, Y. F., & Funk, R. R. (2006). Development and validation of the GAIN Short Screener (GSS) for internalizing, externalizing and substance use disorders and crime/violence problems among adolescents and adults. American Journal on Addictions, 15 Suppl 1, 80-91. doi: 10.1080/10550490601006055

Edlund, M. J., Wang, P. S., Berglund, P. A., Katz, S. J., Lin, E., & Kessler, R. C. (2002). Dropping out of mental health treatment: patterns and predictors among epidemiological survey respondents in the United States and Ontario. American Journal of Psychiatry, 159(5), 845-851.

Emshoff, J. G., Darnell, A. J., Darnell, D. A., Erickson, S. W., Schneider, S., & Hudgins, R. (2007). Systems change as an outcome and a process in the work of community collaboratives for health. American Journal of Community Psychology, 39(3-4), 255-267. doi: 10.1007/s10464-007-9110-7

General Accounting Office. (2008). Young adults with serious mental illness: Some states and federal agencies are taking steps to address their transition challenges. (GAO-08-678). Washington, DC: General Accounting Office (GAO).

Harder, V.S., Stuart, E.A., & Anthony, J.C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15(3), 234-249. doi: 10.1037/a0019623

Ising, H. K., Veling, W., Loewy, R. L., Rietveld, M. W., Rietdijk, J., Dragt, S., . . . van der Gaag, M. (2012). The validity of the 16-item version of the Prodromal Questionnaire (PQ-16) to screen for ultra high risk of developing psychosis in the general help-seeking population. Schizophr Bull, 38(6), 1288-1296. doi: 10.1093/schbul/sbs068

Kessler, R. C., Barker, P. R., Colpe, L. J., Epstein, J. F., Gfroerer, J. C., Hiripi, E., . . . Zaslavsky, A. M. (2003). Screening for serious mental illness in the general population. Archives of General Psychiatry, 60(2), 184-189.

Kessler, R. C., Demler, O., Frank, R. G., Olfson, M., Pincus, H. A., Walters, E. E., . . . Zaslavsky, A. M. (2005). Prevalence and treatment of mental disorders, 1990 to 2003. New England Journal of Medicine, 352(24), 2515-2523. doi: 10.1056/NEJMsa043266

LaPorte, Thomas M., Haber, Mason G., & Malloy, Joanne M. (2014). Wraparound Team Composition, Youth Self-determination, and Youth Satisfaction in Transition Services. The Journal of Behavioral Health Services & Research, 1-19. doi: 10.1007/s11414-014-9434-7

Legere, L., & Nemec, P. B. (n.d.). Integration and isolation of certified peer specialists. Non-published manuscript.

Malloy, J. M., Haber, M. G., Burgess, C. B., LaPorte, T. M., & Corral, N. (2015). Building the Evidence-Base for the RENEW Transition Model: Theoretical & Methodological Developments. Symposium presented at the 27th Annual Children’s Mental Health Research and Policy Conference, Tampa, FL.

McCaffrey, D.F., Ridgeway, G., & Morral, A.R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403-425.

Menard, S. (2009). Victimization & Illegal Behavior. Huntsville, TX: Crime Victim’s Institute, Criminal Justice Center, Sam Houston State University.

Menehan, K. . (2014). Early Detection and Intervention for the Prevention of Psychosis in Adolescents and Young Adults (pp. 43): Robert Wood Johnson Foundation.

Miller, S., Ringeisen, H., Munoz, B., Rohloff, H., Hedden, S., Colpe, L., & Embry, V. (2014). Predictors of mental health service use in young adulthood: Results from the National Survey on Drug Use and Health. Unpublished manuscript.

National Center for PTSD. (2012). Using the PTSD Checklist (PCL). Washington, DC: U.S. Department of Veterans Affairs, VA National Center for PTSD.

Novak, S. P., Colpe, L. J., Barker, P. R., & Gfroerer, J. C. (2010). Development of a brief mental health impairment scale using a nationally representative sample in the USA. International Journal of Methods in Psychiatric Research, 19 Suppl 1, 49-60. doi: 10.1002/mpr.313

Nowell, B. (2009). Profiling capacity for coordination and systems change: the relative contribution of stakeholder relationships in interorganizational collaboratives. American Journal of Community Psychology, 44(3-4), 196-212. doi: 10.1007/s10464-009-9276-2

Nowell, B., & Foster-Fishman, P. G. (2011). Examining multi-sector community collaboratives as vehicles for building organizational capacity. American Journal of Community Psychology, 48(3-4), 193-207. doi: 10.1007/s10464-010-9364-3

Rosenbaum, Paul R., & Rubin, Donald B. (1983). The central role of the propensity score in observational studies for causal effects Biometrika, 70(1), 41-55.

Shadish, William R. (2010). Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods, 15(1), 3-17. doi: http://dx.doi.org/10.1037/a0015916

Shadish, William R., Cook, Thomas D., & Campbell, Donald T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.

Slaghuis, S. S., Strating, M. M., Bal, R. A., & Nieboer, A. P. (2011). A framework and a measurement instrument for sustainability of work practices in long-term care. BMC Health Services Research, 11, 314. doi: 10.1186/1472-6963-11-314

Steadman, H. J., Mulvey, E. P., Monahan, J., Robbins, P. C., Appelbaum, P. S., Grisso, T., . . . Silver, E. (1998). Violence by people discharged from acute psychiatric inpatient facilities and by others in the same neighborhoods. Archives of General Psychiatry, 55(5), 393-401.

Substance Abuse and Mental Health Services Administration. (2010a). Adults with Mental Illness: Findings from the 2008 National Survey on Drug Use and Health. Rockville, MD: (Center for Behavioral Health Statistics and Quality and Center for Mental Health Services, HHS Publication No. SMA 10-4614, Analytic Series A-31).

Substance Abuse and Mental Health Services Administration. (2010b). Results from the 2009 National Survey on Drug Use and Health: Volume I. Summary of National Findings. (HHS Publication No. SMA 10-4586Findings). Rockville, MD: Retrieved from http://www.oas.samhsa.gov.

Sum, A., Khatiwada, I., Trubskyy, M., Ross, M., McHugh, W., & Palma, S. (2014). The plummeting labor market fortunes of teens and young adults. Washington, DC: The Brookings Institution.

Tondora, J., & Miller, R. (2009). Centered Care Planning Questionnaire — Person in Recovery (PIR) Version. New Haven, CT: Yale Program for Recovery and Community Health.

Tracey, T. J., & Kokotovic, A. M. (1989). Factor structure of the Working Alliance Inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 1, 207-210. doi: 10.1037//1040-3590.1.3.207

United Nations. (2014). Mental health matters social inclusion of youth with mental health conditions. Retrieved from http://www.un.org/esa/socdev/documents/youth/youth-mental-health.pdf

Walker, J. S., Koroloff, N., & Mehess, S. J. (2015). Community and State Systems Change Associated with the Healthy Transitions Initiative. The Journal of Behavioral Health Services & Research, 42(2), 254-271. doi: 10.1007/s11414-014-9452-5

Walker, J. S., Thorne, E. K., Powers, L. E., & Gaonkar, R. (2009). Development of a scale to measure the empowerment of youth consumers of mental health services. Journal of Emotional and Behavioral Disorders. doi: 10.1177/1063426609337388

West, Stephen G., Biesanz, Jeremy C., & Pitts, Steven C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H.T. Reis & C.M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 40-84). New York: Cambridge University Press.

Wolman, J., Campeau, P., Dubois, P., Mithaug, D., & Stolarski, V. (1994). AIR Self-Determination Scale and user guide.

Wright, D., Bowman, K., Butler, D., & Eyerman, J. (2005). Non-response bias from the national household survey on drug abuse incentive experiment. Journal of Economic and Social Measurement, 30, 219-231.



LIST OF ATTACHMENTS

1. Community Support for Transitions Inventory (CSTI)

2. State Support for Transition Inventory (SSTI)

3. Collaborative Member Survey

4. Collaborative Self-Assessment

5. Project Director Web Surveys

6. Project Director Telephone Interviews

7. Core Staff Web Survey

8. Multi-Media Project Young Adult Probes

9. Supplemental Youth and Young Adult Interview (SYAI), Baseline

10. Supplemental Youth and Young Adult Interview (SYAI), 12 and 24 Months

11. Services and Supports Inventory

12. Grantee Visit In-Person Interviews and Focus Group Guides

7