OMB_Supporting_Statement-12-19-07

The Mental Health Transformation State Incentive Grant

Cross-Site Evaluation

Supporting Statement

A. Justification

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Mental Health Services (CMHS) has funded the Mental Health Transformation State Incentive Grant (MHT SIG) program to assist State efforts in transforming their mental health systems. This program was created based on the findings of the President’s New Freedom Commission on Mental Health, which indicated the mental health system needed to be transformed to meet six goals:

Goal 1	Americans understand that mental health is essential to overall health.
Goal 2	Mental health care is consumer and family driven.
Goal 3	Disparities in mental health services are eliminated.
Goal 4	Early mental health screening, assessment, and referral to services are common practice.
Goal 5	Excellent mental health care is delivered and research is accelerated.
Goal 6	Technology is used to access mental health care and information.

CMHS awarded MHT SIG grants to Connecticut, Maryland, New Mexico, Ohio, Oklahoma, Texas, and Washington in September 2005 and to Hawaii and Missouri in September 2006. In September 2006, a contract was established to conduct this cross-site evaluation.

The cross-site evaluation project will evaluate the effectiveness of the MHT SIG program for nine State grantees. Thus, CMHS is seeking Office of Management and Budget (OMB) approval for data collection associated with this evaluation:

• Key Personnel Interviews (using discussion guides developed for this program)

• Leadership Surveys (using an existing survey instrument)

• Consumer and Family Member Involvement Interviews (using a form developed for this evaluation)

• GPRA Infrastructure Indicators (entered by States into a web-based data collection system using forms developed for this program)

• Mental Health Provider Interviews (using form developed for this evaluation )

• Recovery and Resilience Measurement Surveys (using existing recovery and resilience instruments as selected by the grantee States, along with modifications of OMB-approved National Outcome Measures—NOMs—questions for CMHS discretionary grant programs).

In addition, the evaluation project will conduct secondary analyses of data available from pre-existing Federal and national databases:

• Statewide data on SAMHSA National Outcome Measures (NOMs) will be obtained from the CMHS Community Mental Health Services Block Grant program.

• Statewide data on additional effects of mental health transformation will be obtained from Federal and national databases, such as the Centers for Disease Control and Prevention’s Behavioral Risk Factors Surveillance System; the Agency for Healthcare Research and Quality’s Medical Expenditure Panel Survey; U.S. Census Bureau databases; existing SAMHSA databases; and surveys conducted by relevant national mental health organizations such as the National Alliance on Mental Illness.

Cooperative agreements for MHT SIGs are authorized under Section 520A of the Public Health Service Act, as amended. The MHT SIG program supports an array of infrastructure and service delivery improvement activities to help grantee States build a solid foundation for delivering and sustaining effective mental health and related services. These grants are unique in that they will support new and expanded planning and development to promote transformation to systems explicitly designed to foster recovery and meet the multiple needs of consumers. These grants are designed to advance the vision and goals of the final report of the President’s New Freedom Commission on Mental Health to transform the Nation’s mental health system.

The national evaluation, conducted under the authority of the Secretary of Health and Human Services through the SAMHSA Administrator, is authorized under Section 501(d)(4) of the Public Health Service Act (42 USC 290aa):

The Secretary, acting through the Administrator, shall assure (sic) that the Administration conduct and coordinate (sic) demonstration projects, evaluations, and service system assessments and other activities necessary to improve the availability and quality of treatment, prevention and related services.

CMHS has determined that this program must meet the requirements of the Government Performance and Results Act of 1993 (P.L. 103–62). GPRA requires all Federal agencies to set program performance targets and report annually on the degree to which the previous year’s targets were met. Agencies are expected to evaluate their programs regularly and to use results of these evaluations to explain their successes and failures and justify requests for funding. To meet the GPRA requirements, SAMHSA must collect performance data.

2. Purpose and Use of Information

The evaluation will determine the success of the MHT SIG program in supporting States’ efforts to transform their mental health systems and promote recovery orientation and improved client outcomes.

The evaluation will provide an overall assessment of the MHT SIG program, not a comparison of the States. Using document reviews and data collection efforts described below, the evaluation will identify the full range of strategies used by grantee States, and identify the barriers States faced, how they were addressed, and which strategies are associated with successful transformation. The evaluation will also determine whether infrastructure changes lead to service changes and whether service changes lead to client outcome changes. It is not expected that statewide client outcome changes will be observed during the grant period.

The evaluation will seek to learn and benefit from the natural diversity in the approaches States use. Individual State efforts will be described in a rich case-study type format. In conducting the overall evaluation, various State approaches able to achieve particular types of transformation results may be compared with those unable to achieve such results. Understanding such differences may benefit State grantees, CMHS, and the field.

The evaluation of the MHT SIG program will serve several purposes:

• It will satisfy items in the OMB Program Assessment Rating Tool (PART) pertaining to performance measurement and independent program evaluation;

• It will satisfy Government Performance and Results Act (GPRA) requirements;

• It will determine the effect of the program on SAMHSA’s National Outcome Measures (NOMs), which are used to assess the impact of all SAMHSA programs;

• It will determine the extent to which the program achieves its ultimate goals of transforming mental health systems and improving consumer recovery and resilience;

• It will provide useful information regarding aspects of transformation that are working and not working to grantees as the program progresses, in order to assist them in making their transformation efforts as useful as possible; and

• It will document factors that contribute to successful transformation in order to inform current and future transformation efforts of other States and SAMHSA.

With direction from CMHS staff and in consultation with the nine grantee States and five consumer/family member consultants, a logic model for the program was developed in January 2007 (Attachment 1). This logic model depicts the hypothesis that infrastructure changes may lead to service changes, which may lead to changes in client outcomes coming after the grant period has ended. The logic model provides the basis for the evaluation plan.

Starting with the second column and moving to the right through the logic model, the specific elements of the cross-site evaluation follow:

1. Transformation strategies will be assessed through a number of data collection efforts. The cross-site evaluators will examine existing documents (e.g., State Comprehensive Mental Health Plans) and reports to collect information, and then will conduct interviews to obtain additional information. In-person site visits will be conducted at all nine States in grant years 3 and 5. Specific subjects for the discussion guides for the State site visit interviews will vary, depending on the contents of the State documents. The specific data collection efforts will include:

• Interviews with key State informants, which will help to determine the effect the MHT SIGs have had on the States’ mental health service systems, to understand why the States have chosen the approaches they have, and to learn about any barriers or needs associated with the grant program (Attachment 2). Interviews will be conducted at two times: during baseline site visits in year 3 of the grant and during follow-up site visits in grant year 5. Interviews will be conducted with 8 key informants from among the State leadership in each of the 9 grantee States.

• During grant years 3 and 5, leadership surveys will be conducted with 15 persons in each State (135 total) to understand the leadership characteristics of persons leading the MHT SIG transformation efforts within the States. The instrument to be used is the Multifactor Leadership Questionnaire by Bass and Avolio (Attachment 3), which is the most validated and efficient measure of transformational leadership.

• Consumer and family member involvement in the States’ transformation efforts will be assessed at statewide and local levels. In grant years 3 and 5, the cross-site evaluators will conduct focus groups during site visits, plus phone interviews with 15 persons in each State (for a total of 135) to determine the nature of consumer and family involvement in grant activities. The focus group facilitation guide/interview discussion guide is provided in Attachment 4.

2. Infrastructure changes will be assessed through the use of 7 GPRA Infrastructure Indicators addressing:

• Policy changes,

• Training of mental health and related staff,

• Financing changes,

• Organizational changes,

• Data collection,

• Statewide consumer and family networks, and

• Service changes.

GPRA definitions are provided in Attachment 5, and the GPRA data collection instructions and forms are provided in Attachment 6. Starting in September at the beginning of their third grant year, States will submit individual GPRA measure data to the cross-site evaluators annually.

3. Service improvements will be assessed, in part, through interviews (Attachment 7) with a structured sample of 26 adult and child/youth providers and professional associations in each State (234 total). The annual interviews will determine the effect, if any, that the MHT SIGs have had on mental health providers and the barriers that impede their ability to meet transformation objectives.

4. Outcomes will be assessed in two ways:

• Statewide data on the SAMHSA National Outcome Measures (NOMs) obtained from the CMHS Community Mental Health Services Block Grant program will be analyzed. The NOMs include data pertaining to:

  • Level of functioning,

  • Employment/school status,

  • Criminal justice system involvement,

  • Living situation,

  • Access by age, gender and ethnicity,

  • Use of inpatient psychiatric facilities,

  • Social connectedness,

  • Self-reported outcomes,

  • Cost effectiveness, and

  • Use of evidence-based practices.

• In addition, each State will collect data on individual recovery (for adults), individual resilience (for children/youth), and system orientation toward recovery at two different times. States may select their own instruments as long as they meet CMHS criteria. Each State will develop its own plan to measure recovery and resilience, including its approach to random sampling of participants. State plans will be reviewed and approved by CMHS prior to implementation. Attachment 8 provides:

• Criteria for recovery, resilience, and system measures

• A list of candidate measures

• Protocols to be followed in collecting these measures; a minimum of 300 persons are to be interviewed at two different times, broken down as follows: 75 adults and 75 children/youth in services expected to be affected by transformation efforts (i.e., the impacted groups) and the same numbers of adults and children/youth in services not expected to be affected by transformation efforts (i.e., the non-impacted groups).

• An estimate of the cost of conducting recovery, resilience, and system orientation interviews.

In addition to the recovery and/or resilience questionnaire, each participant will complete the Individual Interview Guide for adults or children/youth included in Attachment 9. These guides will be used regardless of which recovery and resilience instruments States select. The guides are based on the OMB-approved CMHS discretionary grant NOMs questions (OMB No. 0930-0285, expiration 4/30/2010) with the following modification: due to redundancy of the questions with more complete information to be obtained through the recovery and resilience measures, Sections B (level of functioning), F (perception of care), and G (social connectedness) will not be used and have been removed from the interview guides.

5. Cost impact analysis will include document reviews and interviews to identify the financial impact of the MHT SIG grants on the States’ mental health service system, i.e., how funds were used, what was funded by the grant, and what benefits were created. Sources of cost information will be original and continuing grant applications, other financial information available from CMHS, and financial data provided directly by the States. After review of available data, cross-site evaluators will then develop discussion guides to conduct phone interviews with State staff in the third and fifth grant years to collect other needed data (Attachment 10).

6. Information from national databases will be used to supplement data collected from the States. Attachment 11 shows the databases that may be used.

3. Use of Information Technology

The evaluation project has developed a password-protected Web Extranet site for the MHT SIG cross-site evaluation project: http://systems.hsri.org/mhtsig. This site provides access to

• Transformation Tracker, an application where grantee States can directly enter information for the GPRA Infrastructure Indicators.

• A resources section containing notes for various project meetings, transformation-specific literature references and documents, Web sites for the grantee States, and recorded trainings conducted for the MHT SIG evaluation project

• Contact information for Federal and State staff and other persons associated with the MHT SIG program and the evaluation project

• A consultation database where guidance about various evaluation topics can be viewed

• A component where State site visit information will be entered once available

The capability to enter GPRA information directly in an electronically coded format reduces the burden for both the States and the cross-site evaluators. It also enables State staff to enter GPRA information when it is convenient to them, reducing the perceived burden.

The Web site also allows the Project Officer for the cross-site evaluation, the Federal Project Officers for the individual grantee States, CMHS program consultants, the States themselves, and the Consumer/Family Member Consultants to design and run reports on the GPRA data, providing easier access to the data and reducing the burden that would be needed if all reports had to be created and provided by the cross-site evaluators.

The project Web site will also be set up to accept batch files from the grantee States for data on their recovery and resilience studies. The Web site will enable States to download the data format file (either Excel or Access) to be used, and subsequently to upload data on interviews with non-impacted and impacted groups. States will be able to submit cumulative batch files so the evaluators at the Federal and State levels will have only one file with all of the latest data for each of the data sets. The burden on the States will be reduced because they will not have to re-enter data and because the Web site report generation capabilities will make it easy for them to review their own data to ensure its completeness and accuracy.

The evaluation project will use Web conferencing technology whenever appropriate to provide training on data collection methodology or other topics of importance to the evaluation. The program will place recordings of these trainings on our Web site so that State staff can listen to presentations that they are not able to participate in live; this will increase the accessibility of trainings to State staff. 

4. Efforts to Identify Duplication

The survey instruments and interview protocols used to collect data for the MHT SIG evaluation are unique to this initiative, and the frequency of data collection has been reduced to a minimum. The information from these instruments is needed to determine the success of planning and implementation and the impact of the MHT SIG activities in the grantee States.

In formulating the evaluation plan, the Program has carefully considered how to minimize burden and have included the following approaches to do so:

• To the extent possible, information regarding the transformation process and cost impact will be gathered through review of documents that grantees are preparing and/or submitting to their Project Officers in the normal course of the grant program. Interview questions will then be tailored to minimize the time that project staff must spend to provide supplemental necessary information.

• Impacts on core consumer outcomes will be assessed through secondary analysis of existing NOMs data provided by States through the Community Mental Health Services Block Grant program.

• For the recovery/resilience study, the Program has removed Sections B, F and G of the discretionary NOMs instrument because they are redundant with information collected by the recovery and resilience instruments.

• To assess statewide impact on numerous community outcomes, the Program will obtain and analyze existing data from the following Federal and national databases:

• CMHS Uniform Reporting System for data on statewide inpatient utilization, expenditures, and access

• CMHS Study on Seclusion and Restraint for data on events and injuries

• SAMHSA Treatment Episode Data Set for data on client-level demographics and substance use

• National Alliance on Mental Illness for Grading the States report data on transformation efforts

• CMHS State Data Profiles for data on policy, transformation initiatives, services, and revenue/expenditures

• Centers for Disease Control and Prevention for State data on bad mental health days, substance abuse, and stigma; and the National Center for Health Statistics for data on suicide deaths

• Agency for Healthcare Research and Quality for data on mental health and prescription drug spending

• American Psychiatric Association, American Psychological Association, and National Association of Social Workers for providers per 100,000 data

• U.S. Census Bureau for population, income, and employment data

5. Involvement of Small Entities

Most of the data collection will be coming from State agency staff. The Program will not be collecting data from a substantial number of small entities.

6. Consequences If Information Collected Less Frequently

Failure to collect the information on this proposed schedule would prevent the MHT SIG program from meeting its obligations to determine the impact of the MHT SIG program. Most of the data is collected only twice (e.g., during the third and fifth grant years for State agencies, or at baseline and 12-month follow-up for the recovery and resilience measurements). Two measurements are needed to identify changes over time. The only exceptions are GPRA data and mental health provider data, which are collected annually during the third, fourth, and fifth grant years. The Government Performance and Results Act of 1993 (P.L. 103-62) requires that performance targets be set annually and that reports be provided annually regarding the degree to which the previous year’s targets were met. Provider interviews are being conducted annually because of the critical role that providers play in translating State transformation efforts into improved consumer outcomes.

7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The 60-day notice required by 5 CFR 1320.8(d) was published in the Federal Register on August 22, 2007 (72 FR 47056). No comments were received in response to this notice.

The evaluation plan was developed in consultation with the nine grantee States; and five cross-site evaluation Consultants who represent youth, adult, and older adult consumers; and family members of youth and adult consumers. Evaluators and other staff from the grantee States and the Consultants participated in monthly Evaluation Work Group calls to discuss the development and refinement of the evaluation plan. Feedback on aspects of particular importance to consumers and family members (e.g., measurement of recovery and resilience and consumer and family involvement in transformation) was also solicited from consumer, youth, and family participants in the MHT SIG program through the program’s “Consumer, Youth, and Family” listserv.

In addition to review by grantee States and cross-site evaluation Consultants, the draft evaluation plan was reviewed by, discussed in depth with, and modified in accordance with feedback from three senior evaluators who have worked on other CMHS cross-site evaluations:

• Carol Bianco, Managing Director, Advocates for Human Potential, Two E-Comm Square, 324 Broadway, Albany, NY 12207. Phone number is 518-475-9146, extension 226.

• Howard Goldman, MD, PhD, Professor of Psychiatry at the University of Maryland School of Medicine, 655 West Baltimore Street
  Baltimore, Maryland 21201. Phone number is 410-646-1740.

• Hank Steadman, President, Policy Research Associates, 345 Delaware Avenue, Delmar, New York 12054. Phone number is 518-439-7415.

9. Payment to Respondents

No payments or gifts will given or offered to respondents by the cross-site evaluation project, other than nominal payments to individuals who participate in the consumer/family member involvement interview. Payments to consumer and family members will be provided to ensure the range of participation necessary to allow the cross-site evaluation to independently assess their involvement at the State and local levels. Without such payments, it is much less likely that key consumer and family members will be willing to participate in focus groups and interviews lasting up to 90 minutes.

Grantee States will make their own decisions about whether to provide compensation to persons who participate in their recovery and resilience measurement interviews. In estimating costs, the cross-site evaluation has estimated that participants would be paid $25 for both baseline and follow-up interviews in order to access this hard-to-reach population.

10. Assurance of Confidentiality

Protecting the security of data is paramount for any evaluation project, particularly when information is collected about identifiable individuals. The cross-site evaluation team, which has broad experience conducting national and cross-site evaluations, has developed a comprehensive approach and will diligently follow it to ensure the security of all data collected by the MHT SIG cross-site evaluation. All project staff will be trained in data security policies (staff responsibilities for securing hard-copy materials and computer workstations, shredding discarded copies of documents, maintaining protection of information collected, etc.). These policies will be shared across the project team and reinforced through training as needed.

Because the statewide NOMs and other national information come from existing national databases, no security or privacy issues exist for them. The evaluation project will use the following approaches to ensure the security and privacy of other data collection efforts:

• GPRA Infrastructure Indicators—This component of data collection does not involve collection of information about identifiable individuals and, therefore, involves no information requiring security or privacy precautions.

• Recovery and resilience—Individual grantee States will be expected to submit their plans for the recovery and resilience component of the evaluation to a local Institutional Review Board (IRB), or to produce evidence from the IRB that submission is not required. All States will be required to obtain written informed consent from all participants in this component of the evaluation. Grantee States may choose whether or not to obtain a Certificate of Confidentiality for this component of the evaluation. The cross-site evaluation project will not accept any electronic or hard-copy files that have personally identifiable information. Grantee states will need to develop a coding scheme to suppress personally identifiable information on respondents. To ensure that the cross-site evaluation project does not get personally identifiable information, grantee States will be required to describe in detail the data format they plan to submit to the cross-site evaluation before they actually submit any data. Further, all reporting will aggregate State data into adult and child/youth groups.

• Leadership surveys—These surveys will ask about the perception of different aspects of leadership for the State transformation project. Leaders will rate themselves, and raters (e.g., Transformation Work Group members) will assess the grant project leadership. A limited number of cross-site evaluation project staff will have access to the actual names of persons who participate in this survey, and response forms will be coded when sent out to respondents. This means that names of respondents will not be shown on the forms used by staff doing data entry and analysis. Further, all information will be aggregated by State, so individual responses will not be identifiable. All reports about the interviews will only provide information for all nine States collectively, and will be written so it will not be possible to identify any individual or State.

• Mental health provider interviews—These interviews will ask about the impact of State transformation grant activities on providers. Responses are not seen as containing private information, and results will be aggregated by State so individual provider answers generally will not be identifiable. When only one type of provider falls in a particular category, e.g., a youth inpatient provider, then aggregation will not be possible.

• State agency interviews (including the cost impact interviews) —These interviews will discuss the planning, implementation, and results of transformation grant activities. Responses are not seen as containing private information, but they will be aggregated to the extent possible so individual answers will not be identifiable. In the case of the cost impact interviews, aggregation will not be possible since there will be only one respondent per State. If audiotaping (with interviewees’ permission) is done during these interviews, the audio records will be erased after the interview notes have been created and verified.

• Consumer and family member involvement interviews—These interviews will be conducted using focus groups and individual interviews. For the focus groups, participants will have to sign an informed consent form that includes an agreement not to disclose any personal information that might come out during the focus group discussion. In the case of individual interviews, a person will be read an informed consent statement, and their participation will indicate their agreement with it. No personal names will be shown on the handwritten notes taken during the focus group or interviews. If audiotaping (with interviewees’ permission) is done, the audio records will be erased after the focus group/interview notes have been created and verified.

Summary

All data collection instruments and data collection procedures have been carefully constructed to avoid any potential issues with data that may raise protection or privacy concerns. All physical documents containing information about identifiable individuals (e.g., faxes, handwritten surveys, notes) will be stored in a secure central location by Cross-site evaluation staff charged with their safekeeping. For any such data stored electronically, user IDs and passwords will be required for access.

Both the Web server and project databases will reside in a staffed data center and will have firewall protection. The database will not allow anonymous connections, and account information will be encrypted. If a security incident occurs, proper incident response procedures will be followed. Supervisors are responsible for ensuring that all project staff observe all security requirements and receive appropriate security training. Reports and publications from these data will be limited to aggregate data analysis that fully protects the identity of individual participants.

11. Questions of a Sensitive Nature

With the exception of the Recovery and Resilience component of the evaluation, respondents will not be asked any questions of a personally sensitive nature. The remainder of the interviews and surveys are limited to questions regarding perceptions of grant planning and implementation activities and results among key stakeholders of the grants.

The Recovery and Resilience component of the evaluation is critical for demonstrating that the MHT SIG program has achieved the primary goal of transformation—facilitating the recovery and resilience of individual consumers. Grantee States will be expected to submit their individually developed plans for the recovery and resilience component of the evaluation to a local Institutional Review Board (IRB), or to produce evidence from the IRB that submission is not required. All States will be required to obtain written informed consent from all participants in this component of the evaluation, using locally developed consent forms that adhere to the requirements of the local IRB. For data collected from minors, written parental consent will be required.

The cross-site evaluation will not collect respondent social security numbers.

12. Estimates of Annualized Hour Burden

Table 1 provides the basis of the resulting estimates of the hour burden of collection of information, based on the proposed protocols and instruments. The bases for these burden estimates are as follows:

• Recovery and resilience—Times for some instruments were provided by the developers; times for other instruments were estimated based on the number of items they contained as compared with instruments with known administration time.

• Leadership surveys—The estimated time comes from the survey owner.

• Provider interviews—The estimated time is based on the response time for four questions.

• GPRA Infrastructure Indicators—It has been estimated that on average States will need 15 minutes to enter information for each of the 7 Indicators and will have 48 measures to enter. The time required to develop the individual GPRA target activities is a program requirement, not a requirement of the cross-site evaluation itself.

• Consumer/family member involvement interviews—The time is based on the time allocated for focus groups (90 minutes) and results of a pilot test of the instrument using focus group and individual interview approaches.

• State Agency interviews—The estimated time is the average of time allocated for in-person interviews occurring during State site visits.

• Cost impact interviews—The estimate is based on 1 hour for data collection and half an hour to respond to four questions during the phone interview.

The hourly rates in Table 1 are based on the following:

• Adult consumers and consumer/family member respondents (Recovery and system recovery orientation; Consumer/family involvement; and 4 of 15 respondents to Leadership survey)—Average of 2006 grantee State median wage rates for all occupations from the U.S. Department of Labor Bureau of Labor Statistics

• Child/youth consumers (Resilience)—Average of minimum wage rates for the period April 2008 to April 2011

• State data entry staff (GPRA Indicators), mental health program managers (11 of 15 respondents to Leadership survey; State agency staff interviews; Cost impact; and 23 of 26 respondents to Provider interviews), and associations executives (3 of 26 respondents to Provider interviews)—Average of 2006 median wages for grantee States from the Department of Labor Occupational Information Network (O*NET) obtained on October 3, 2007

  • For each State, the total of 15 respondents to the Leadership survey consists of 11 State mental health program managers at $36.28/hour and 4 consumer/family members at $14.75/hour, for a weighted average of $30.54/hour.

  • For each State, the total of 26 respondents to the Provider interviews consists of 23 mental health program managers and 3 association executives at $62.14/hour, for a weighted average of $39.26.

Based on the expected number of respondents and burden hours, the estimate for the total annual cost burden to respondents is $23,334.

Table 1. Estimate of Total Annual Response Burden and Associated Cost

Instrument	No. of States	No. of Respon- dents/ State	Total no. of respon-dents	No. of Responses/ respondent	Average burden/ response (hours)	Total Annual burden (hours)	Hourly Rate	Total Annual Cost
YEAR 1 (2008)
Recovery (non-impacted)	7	75	525	1	0.5	262.5	14.75	$3,872
Resilience (non-impacted)	7	75	525	1	0.6	315	6.86	2,161
Leadership survey	7	15	105	1	0.33	34.7	30.54	1,060
Provider interviews	7	26	182	1	0.5	91	39.26	3,573
GPRA Indicators	7	1	7	1	12	84	11.51	967
Consumer/family involvement	7	15	105	1	2	210	14.75	3,098
State agency staff interviews	7	8	56	1	1.13	63.3	36.28	2,297
Cost impact	7	1	7	1	1.5	10.5	36.28	381
Subtotal (year 1)			1512			1070.9		$17,407
YEAR 2 (2009)
Recovery (impacted)	7	75	525	1	0.5	262.5	14.75	$3,872
Recovery (non-impacted)	2	75	150	1	0.5	75	14.75	1,106
Recovery & system recovery orientation (non-impacted)	7	75	525	1	1	525	14.75	7,744
Resilience (impacted)	7	75	525	1	0.6	315	6.86	2,161
Resilience (non-impacted)	9	75	675	1	0.6	405	6.86	2,778
Leadership survey	2	15	30	1	0.33	9.9	30.54	302
Provider interviews	9	26	234	1	0.5	117	39.26	4,593
GPRA Indicators	9	1	9	1	12	108	11.51	1,243
Consumer/family involvement	2	15	30	1	2	60	14.75	885
State agency staff interviews	2	8	16	1	1.13	18.1	36.28	657
Cost impact	2	1	2	1	1.5	3	36.28	109
Subtotal (year 2)			2721			1898.5		$25,450
YEAR 3 (2010)
Recovery (impacted)	2	75	150	1	0.5	75	14.75	$1,106
Recovery & system recovery orientation (impacted)	7	75	525	1	1	525	14.75	7,743
Recovery & system recovery orientation (non-impacted)	2	75	150	1	1	150	14.75	2,213
Resilience (impacted)	9	75	675	1	0.6	405	6.86	2,778
Resilience (non-impacted)	2	75	150	1	0.6	90	6.86	617
Leadership survey	7	15	105	1	0.33	34.7	30.54	1,060
Provider interviews	9	26	234	1	0.5	117	39.26	4,593
GPRA Indicators	9	1	9	1	12	108	11.51	1,243
Consumer/family involvement	7	15	105	1	2	210	14.75	3,098
State agency staff interviews	7	8	56	1	1.13	63.3	36.28	2,297
Cost impact	7	1	7	1	1.5	10.5	36.28	381
Subtotal (year 3)			2166			1788.4		$27,129
AVERAGE			2133			1585.9		$23,329

Note: Totals may not add exactly due to rounding.

13. Estimates of Annualized Cost Burden to Respondents

This data collection effort involves no capital, start-up, maintenance, nor operational costs for the cross-site evaluation of the MHT SIG program.

14. Estimates of Annualized Cost to the Government

The annual cost to the Government of the proposed data collection consists of 20 percent of the Government Project Officer’s salary (grade 14, step 5), and 100 percent of a contract awarded for the conduct of the MHT SIG cross-site evaluation by the U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. The estimated average annual cost of these expenses is $739,924.40 per year.

The National Institute of Mental Health transfers $50,000 annually to SAMHSA to support this evaluation.

15. Change in Burden

This is a new project.

16. Time Schedule, Publication and Analysis Plans

16a. Time Schedule

Table 2 shows the time schedule for implementing and using the proposed instruments. A 3-year clearance is requested for this project.

Table 2. Schedule for Use of MHT SIG Cross-Site Evaluation Instruments

Activity	Date
OMB approval	Spring 2008
Data collection begins	As soon as OMB approval is received
Data collection ends	Spring 2011
Data analysis	Ongoing up through Summer 2011
Completion of Report	Summer/Fall 2011

16b. Publication Plans

The evaluation contract for the grant program anticipates that aggregate results from the national evaluation will be incorporated in text and charts of the following publications, planned for completion and distribution in 2011:

• An Executive Summary of the evaluation of the MHT SIG grant program; and

• A Final Evaluation Report describing the data collection, analysis, and findings on what approaches were found to be successful in promoting the transformation of State mental health systems; case studies for each State also will be part of this report.

SAMHSA may also choose to incorporate the aggregate results from the cross-site evaluation in journal articles, scholarly presentations, and congressional testimony on outcomes resulting from the MHT SIG grant program.

16c. Analysis Plan

Our evaluation will be asking a series of questions that pertain to constructs and relationships shown in the logic model. Generally, the questions address whether MHT SIG States complied with MHT SIG requirements as identified by SAMHSA (Questions 1–3) and whether expected associations between constructs were observed (Questions 4–6). Other questions will be asked that address the effectiveness of the MHT SIG as conventionally measured in evaluations (e.g., Question 7) and questions related to the process of implementing transformation (e.g., Question 8). Examples of the questions to be addressed include the following:

Controlling for State and Federal inputs:

1. Did the MHT SIG States make infrastructure changes that are consistent with the goals and recommendations of the New Freedom Commission?

2. Did the MHT SIG States make service system changes that are consistent with the goals and recommendations of the New Freedom Commission?

3. Did measures of consumer outcomes, including recovery, improve in MHT SIG States?

4. Are measures of infrastructure change associated with changes in consumer outcomes?

5. Are measures of infrastructure change associated with measures of service improvements?

6. Are measures of service improvement associated with changes in consumer outcomes?

7. Do consumers and families in MHT SIG States experience better outcomes than consumers and families in comparable non-MHT SIG States?

8. Do States that engage in more transformative practices demonstrate more infrastructure development?

The general analysis approach will be to do a stepwise progression through the data:

• Exploratory analyses using both quantitative (means, ranges, standard deviations) and qualitative (coding of main themes from interviews of consumer/family members)

• Relational analyses within States to match up qualitative themes and findings with quantitative data

• Cross-site analyses to identify patterns of quantitative and qualitative findings as the basis for findings about the overall MHT SIG program

17. Display of Expiration Date

The expiration date will be displayed on all instruments.

18. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Six components of the evaluation will involve primary data collection:

Interviews with key State informants

The potential respondent universe for the State informant interviews is all State agency heads within the grantee States. In order to limit burden and impose consistency across the grantee sites, interviews will be conducted only with the same 8 major stakeholder positions identified in the original MHT SIG grant announcement in each of the 9 grantee States:

• Project Director

• Transformation Working Group Chair

• Mental Health Agency Director

• Medicaid Agency Director

• Education Agency Director

• Criminal/Juvenile Justice Agency Director

• Housing Agency Director or Senior Staff

• Employment Agency Director or Senior Staff

Leadership surveys

The potential respondent universe for the leadership survey could be conceived in many different ways. The Program has chosen to focus the respondent pool on those who are in most direct contact with the leaders to be evaluated (that is, the MHT SIG Project Director and Transformation Working Group Chair or Co-chairs for each State). Respondents will be Transformation Working Group (TWG) Subcommittee Chairs and Consumer/Family Members on the TWG or its subcommittees. Depending on the number of TWG Chairs and Subcommittees in each State, up to 15 interviews will be done for each State. Because the concept of a consumer-driven system is central to transformation, the raters will be selected so that at least one-third will be consumer/family members.

Consumer and family member involvement

For this component, the potential respondent universe is consumers, both current and former, of State mental health systems, and family members of current and former consumers. The evaluators will work to ensure that the samples drawn (to the degree possible) are representative of the State population in terms of culture, age, and locale. In order to minimize burden and maximize how representative the information provided by each respondent is, the respondents recruited will be those who can best speak to the involvement of consumers and family members from their community in the transformation process. Sources of potential candidates will include:

• Consumers and family members serving on the State Transformation Working Groups

• Consumers and family members serving as MHT SIG representatives of consumer or family organizations in some other capacity

• Consumers and family members serving in any other working or advisory capacity in State projects

• Consumers and family members identified through other means (e.g., through providers, advocacy groups)

• Representatives of the State affiliates of the National Coalition of Mental Health Consumer/Survivor Organizations (NCMHCSO)

• Representatives of any other statewide consumer organization active in the State

• Representatives of the State affiliates of the National Alliance on Mental Illness (NAMI)

• Representatives of the State affiliates of the Federation of Families for Children’s Mental Health

• Representatives of any other statewide family organization in the State

• Representatives of the State Mental Health Authorities’ (SMHAs’) offices of consumer affairs

• Representatives of the SMHAs’ office of multicultural affairs

• Representatives of any Tribal Nations within the States

• Representatives of the State Protection and Advocacy for Persons With Mental Illness (PAIMI) agency

The names and contact information for potential candidates from each State will be provided to the cross-site evaluators who will recruit participants, schedule and conduct focus groups and interviews, and monitor progress. A total of 15 individuals will be interviewed per State in grant years 3 and 5. It is anticipated that the majority of these individuals will fall into at least one of the above categories. In addition, each sample will be designed to include at least five individuals from each of the following groups:

• Those who are involved in State or local transformation initiatives

• Those who are not involved in State or local transformation initiatives (if needed, up to two of these participants may instead be key informants believed to have knowledge of the experiences of consumers, youth and family members who are not involved in State or local transformation initiatives)

• Those who are involved in consumer, youth, or family organizations

After doing initial assessment interviews to ensure the appropriateness of the potential respondents, 15 individuals will be sampled in each State to participate in either a focus group (first choice) or individual interview (alternate choice). In each State, the cross-site evaluators will select individuals who are willing to provide information about consumer and family involvement.

GPRA Infrastructure Indicators

The full universe of 9 MHT SIG grantees will report data on the GPRA Infrastructure Indicators. This is consistent with the performance monitoring intent of the GPRA legislation.

Provider Interviews

The potential respondent universe for the provider interviews is all mental health and related service providers in the grantee States. To keep within our budget and to limit burden, the Program will interview just 26 providers per grantee State, sampling from the critical provider categories below:

• Inpatient service providers – adult (3) and children/youth (1)

• Residential service providers – adult (4) and children/youth (2)

• Outpatient service providers – adults (5) and children/youth (5)

• Emergency service providers (3)

• Professional associations (3)

Recovery and resilience

For the recovery and resilience component, the potential respondent universe is the people receiving mental health services funded by the State. Each State will identify services that have been or are anticipated to be impacted by the MHT SIG grant—one or more programs for adults with serious mental illness, and one or more for children/youth with serious emotional disturbance. The State will also identify services that are not anticipated to be impacted by the grant—again, one or more programs for adults, and one or more for children/youth to serve as comparison groups to the impacted groups.

For the service programs chosen by each State, adult and child/youth consumers (and family members for children/youth, as necessary) will be randomly selected from the pool of consumers entering that service program for the first time. Replacement sampling will be employed to achieve full samples at baseline. States will enroll 75 adults and 75 children/youth from impacted services and 75 adults and 75 children/youth in the non-impacted services.

Each State will be responsible for developing their own approach to measurement of recovery and resilience, and their plan will be reviewed and approved by the CMHS staff member leading the cross-site evaluation. Each State will be responsible for drawing their own samples, and monitoring progress and compliance with the objectives of this component.

The cross-site evaluation has determined that total sample size for all States (2,700 persons) will be large enough to produce a 95 percent confidence interval with a margin of error of +/− 10 percent for a comparison of the impacted and non-impacted groups.

2. Information Collection Procedures

Interviews with key State informants and focus groups with consumers and family members about consumer and family involvement will be conducted by cross-site evaluation staff during 2-day in-person site visits in years 3 and 5 of each grant. Four staff members will go on each site visit: two members of the core cross-site evaluation team and two consumer/family member consultants. Persons who cannot participate in the focus groups will be interviewed by phone within two months of the site visit.

Leadership survey data will be collected through phone interviews using the Multifactor Leadership Questionnaire by cross-site evaluation staff. Likewise, provider interviews will be conducted via phone by the cross-site evaluators.

Grantee States will enter data pertaining to the GPRA Infrastructure Indicators directly into Transformation Tracker, an electronic application available through the project’s password-protected Web Extranet site, on at least an annual basis.

For the recovery and resiliency component, States will select the approach they want to use to collect data. Interviews can be done by in-person/phone interviews, mail surveys, or Internet surveys (although in-person interviews are likely to be necessary to get high response rates for the follow-up). Recovery, resilience, and system orientation data will be compiled by the States in a designated format and provided to the cross-site evaluators in batch file or another electronic format.

2. Methods To Maximize Response Rates

The data collection instruments have been developed to minimize the response burden and to increase the likelihood of response. Sampling procedures will further reduce burden on respondent providers. Repeated sampling will be done, if necessary, to reach the target numbers. Follow-up will be done intensively, and interviews will be scheduled at the convenience of the respondent and are expected to be conducted in person. To help States achieve follow-up rates of at least 75 percent, the cross-site evaluators have arranged for two specific trainings on this topic by well-known researchers, one in October 2007 and another in November 2007. These trainings will identify tracking approaches States can use to achieve high follow-up rates. If States provide nominal payments to respondents, this too can help them achieve better follow-up rates.

States will be able to choose their own recovery, resilience, and system orientation measures as long as they meet CMHS criteria. This collaborative approach means that each State will not only be able to choose the services evaluated in the recovery and resiliency component but also will be able to choose the instruments best suited to their own needs. States must provide their recovery and resilience information to the cross-site evaluation as a requirement of their MHT SIG grant. The States will be responsible for working with their participating providers regarding the collection and reporting of data.

4. Tests of Procedures

The evaluation team sought instrument and measurement questions with good psychometric properties for use in this study. For the recovery and resiliency component, a number of measures with excellent face validity and good reliability were identified as appropriate candidates for the States to consider. Many of the proposed instruments are used in studies cited in peer-reviewed journals. Attachment 12 shows the psychometric properties of candidate instruments States can consider. As noted earlier, States can choose any instrument that meets CMHS criteria.

The family and consumer involvement questionnaire was developed using information from the following sources:

• CMHS National Advisory Council Subcommittee on Consumer/Survivor Issues’ definition of consumer driven

• Federation of Families for Children’s Mental Health definition of family driven

• SAMHSA Systems of Care definition of youth guided

• Families America parent questionnaire from the How Are We Doing evaluation system

The questionnaire was developed in collaboration with the grantee States and the Consumer/ Family Member Consultants, and with review by participants on the MHT SIG Consumer, Youth and Family listserv and the CMHS Consumer Advisor/Consultant for the MHT SIG program. Further, a pilot test of the instrument was conducted in late September and early October 2007 to help ensure the questions are clear and provide useful information to assess consumer and family member involvement at the State and local levels (Attachment 13).

5. Statistical Consultants

This statistical section has been reviewed by:

Crystal Blyler, PhD, Project Officer

Substance Abuse and Mental Health Services Administration

Center for Mental Health Services

One Choke Cherry Road, Room 6-1009

Rockville, MD 20857

Phone: 240-276-1910

Fax: 240-276-1910

E-mail: [email protected]

Paul Brounstein, Ph.D.

MANILA Consulting Group

6707 Old Dominion Drive, Suite 315

McLean, VA 22101

Phone: 571-633-9797 x234

Fax: 571-633-1895

E-mail: [email protected]

Andrew Rose, Ph.D.

MANILA Consulting Group, Inc

6707 Old Dominion Drive, Suite 315

McLean, VA 22101

Phone: 571-633-9797 x242

Fax: 571-633-0335

E-mail: [email protected]

Steve Leff, Ph.D.

Human Services Research Institute

2336 Massachusetts Avenue

Cambridge, MA 02140

Phone: 617-844-2507

Fax: 617-492-7401

E-mail: [email protected]

Jim Westrich, M.A.

Human Services Research Institute

2336 Massachusetts Avenue

Cambridge, MA 02140

Phone: 617-844-2524

Fax: 617-492-7401

E-mail: [email protected]

List of Attachments (Submitted Separately)

Attachment 1: Logic Model

Attachment 2: Discussion Guides and Associated Protocols for In-Person Site Visits

Attachment 3: Sample Page from Multifactor Leadership Questionnaire

Attachment 4: Adult, Youth, and Family Member (AYF) Screening Questions, Semi-Structured Focus Group Facilitation/Interview Guide, and Questionnaire

Attachment 5: MHT SIG GPRA Infrastructure Indicator Definitions

Attachment 6: GPRA Infrastructure Indicator Data Collection Instructions and Forms

Attachment 7: Discussion Guide and Protocols for Provider Interviews

Attachment 8: Measurement of Recovery, Resilience, and System Orientation

Attachment 9: Individual Interview Guides for Adults, Caregivers of Children and Youth, and Children/Youth

Attachment 10: Discussion Guide and Protocols for Interviews on Cost Impact

Attachment 11: Other National Databases that Might Be Used

Attachment 12: Candidate Instruments for Recovery and Resilience Measurement and Their Psychometric Properties

Attachment 13: Summary of Results for the Pilot Test of Consumer/Family Member Involvement Survey Instrument