Independent Evaluation of the Community Mental Health Services Block Grant
Supporting Statement
A. JUSTIFICATION
1. Circumstances of Information Collection.
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) requests clearance from the Office of Management and Budget (OMB) for a set of interview protocols and surveys of key Federal, State, and community stakeholders involved in the implementation and oversight of the Community Mental Health Services Block Grant (CMHS BG). These instruments will be used to gather data about CMHS BG processes and outcomes that can be used to evaluate the extent to which the CMHS BG is meeting its intended goals and the means by which it is doing so. This data collection effort constitutes the backbone of the first independent evaluation of the CMHS BG program. The CMHS BG Program is authorized under sections 1911 to 1920 of the Public Health Service Act.
History and Legislative Requirements
The CMHS BG Program was originally created by the Omnibus Budget Reconciliation Act of 1981 (Public Law 97-35), as one of several block grant programs introduced to provide a formula-based distribution of Federal funds to States, increase their flexibility, and reduce the administrative burden on the Federal Government. This block grant was part of a larger effort to strengthen the Federal effort to combat drug abuse, alcohol abuse, and mental illness by (1) reorganizing the Alcohol Drug Abuse and Mental Health Administration (ADAMHA); (2) authorizing or reauthorizing a series of prevention and treatment services programs such as the Alcohol, Drug Abuse, and Mental Health Services Block Grant—the precursor to the CMHS BG program; and (3) authorizing a medications development research initiative.
Several major legislative actions have taken place since the Alcohol and Drug Abuse and Mental Health Service Block Grant was initiated in 1981. From 1982 to 1989, various legislative modifications were made to the block grant program. The 1992 ADAMHA Reorganization Act assigned responsibility for administering the CMHS BG program to the Substance Abuse and Mental Health Services Administration (SAMHSA)—a services agency created when the three research institutes housed in ADAMHA were transferred to the National Institutes of Health in 1991. The act also created a separate block grant for substance abuse prevention and treatment—the Substance Abuse Prevention and Treatment Block Grant. Within SAMHSA, CMHS is authorized to administer the CMHS BG.
The Community Mental Health Services Block Grant was funded by Congress to develop community-based systems of care for adults with serious mental illness (SMI) and children with severe emotional disorders (SED), and has been the largest Federal program dedicated to improving community mental health services; in FY2006, the budget was $428 million. States have latitude in determining how to spend their funds to support services for adults with SMI and children with SED. The only requirements outlined in the authorizing legislation for State receipt of CMHS BG funds are provisions to increase children’s services, create a State mental health planning council, and to develop a State mental health plan to be submitted to the Secretary of Health and Human Services. The State mental health planning council is to comprise various State constituents including providers, administrators, and mental health services consumers. Each State plan must:
Provide for the establishment and implementation of an organized community-based system of care for individuals with mental illness
Estimate the incidence and prevalence of adults with SMI and children with SED within the State
Provide for a system of integrated services appropriate for the multiple needs of children
Provide for outreach to and services for rural and homeless populations
Describe the financial and other resources necessary to implement the plan and describe how the CMHS BG funds are to be spent.
In addition, Congress included a maintenance-of-effort (MOE) requirement that a State’s expenditures for community mental health services be no less than the average spent in the two preceding fiscal years.
2. Purpose and Use of Information.
CMHS will conduct an independent evaluation of the CMHS Block Grant Program. In this evaluation, a multi-method evaluation approach is being used to examine Federal and State performance with regard to the CMHS BG and its identified goals. This approach emphasizes a qualitative and quantitative examination of both the CMHS BG process and system-level outcomes. In this approach, Federal and State stakeholder perspectives on the CMHS BG, as captured through semi-structured interviews and surveys, are corroborated and compared to the considerable amount of already-collected source documents provided by States and CMHS (e.g., State plans, implementation reports, review summaries and monitoring site visit reports). More specifically, data collection will be conducted using four primary strategies: interviews and surveys of key stakeholders, data abstraction from source documents (i.e., CMHS BG applications and implementation reports), secondary data analysis (e.g., analysis of Uniform Reporting System (URS) data and National Outcome Measures (NOMS), and case studies highlighting important themes and issues relating to State CMHS BG implementation.
This evaluation is also seeking to measure the effectiveness of the CMHS BG through a variety of infrastructure indicators and NOMS measures. Infrastructure refers to the resources, systems, and policies that support the nation’s public mental health service delivery system, and is a potential contributor to significant State behavioral health system outcomes. Examples of infrastructure include staff training, consumer involvement in the State mental health system, policy changes, and service availability. Outcomes related to infrastructure and the NOMS were included in the program logic model that has been developed and are expected to be examined through the data collection strategies listed above. Logic models are used to provide a comprehensive framework of a program including the program’s inputs or resources, activities, outcomes, and long-term impacts. The overarching evaluation questions developed for this evaluation are based on the CMHS BG program’s logic model, which is included in Attachment A.
Infrastructure indicators that can be measured in this evaluation, for which some form of data can be identified and collected include:
Range of available services within a State
Capacity (# of persons served)
Specialized services (such as co-occurring disorders)
Number of persons served by evidence-based practices (EBPs)
Staff credentialing (identify patterns)
Program accreditation (as a quality marker)
Staff/workforce development (TA & training available for State staff)
Connections with other agencies(e.g., MOUs, joint funding, joint appointments)
Policy changes initiated
Policy changes completed
Consumer involvement
Data will be collected using two primary strategies: 1) on-site interviews with Federal and State staff, and 2) web-based surveys for Planning Council Members, Monitoring Site Visitors, and Regional Reviewers.
Interviews will be conducted with Federal staff involved in the administration of the CMHS BG and State staff from all States and Territories involved in their State’s implementation of the CMHS BG program. The two interview guides, one for Federal staff and one for State staff, range from 54 to 94 open-ended questions. The Federal staff interview is expected to take one hour to complete while the State staff interview is expected to take approximately 3.5 hours to complete. Based on feedback from State stakeholders, the State staff will be interviewed in person by trained project members. Because of the relatively small number of Federal and State staff participating in the evaluation, interviews are an optimal data collection strategy to gather the extensive qualitative data needed for the evaluation while minimizing reporting burden. Federal staff stakeholders also will be interviewed in person due to their close proximity to the interviewers. State Mental Health Agency (SMHA) Commissioners will select those State staff who are knowledgeable about the CMHS BG for participation in the interviews. It is anticipated that, at a minimum, a State Planner, State Data Analyst, and the SMHA Commissioner will participate.
The four (4) web-based surveys will be distributed nationally to:
State Planning Council Chairs
State Planning Council Members
CMHS BG Regional Reviewers
CMHS BG Monitoring Site Visitors
Regional reviewers are individuals knowledgeable about the CMHS BG program who review States’ applications to ensure that they are compliant with the CMHS BG legislation. Monitoring site visitors are consultants to CMHS who provide on-site verification and auditing of States’ system of care to ensure that their system conforms to what was reported in the State’s CMHS BG application. The web-based surveys will be tailored so that each of the four different stakeholder groups will receive survey questions designed to capture their specific knowledge of and experience with the CMHS BG. It is estimated that stakeholders will need one hour to complete their survey, which contains a range of 22 to 42 mostly fill-in-the blank type questions. Each member of the four major stakeholder groups will submit their responses to the survey online over a three-week period.
SAMHSA will analyze the qualitative information and quantitative data collected using these instruments to determine whether and how the CMHS Block Grant program is meeting its intended administrative and substantive goals. The data collection instruments touch on a wide range of topics in two main domains: 1) implementation/process and 2) outcomes. The results of the evaluation will help CMHS to meet its PART requirements, but more importantly, will provide a scientific and comprehensive overview of Federal and State administration and implementation of the CMHS Block Grant. The surveys and interviews will enable CMHS to ask States specific questions about how they implement their mental health systems of care, especially those funded by the CMHS BG, to ensure that the statutorily required services are being provided and identify potential outcomes.
3. Use of Improved Information Technology.
The Project Team has developed a web-based survey instrument that will be used to administer the survey portions of the project’s data collection via the internet. This system is compliant with all SAMHSA ADP-IT requirements and is customizable so that CMHS can use this system as an ongoing evaluation tool. Screenshots of the web-based survey system have been created and included as part of this clearance package (see Attachment A). The web-based survey system contains an easy-to-use user interface and a secure e-mail function to alert stakeholders when they can access the survey. The contractor staff has thoroughly tested the virtual survey system to ensure optimal functioning and compliance to SAMHSA IT and other Federal guidelines.
4. Efforts to Identify Duplication.
The surveys and interviews are specific to this program and the data required are not available anywhere else. The Project Team developed the interview protocols and surveys after a careful review of CMHS BG applications and implementation reports to ensure that the data collection instruments would not duplicate information that could be gathered from these secondary sources.
5. Involvement of Small Entities.
This study does not involve the collection of data from small businesses or other small entities.
6. Consequences of Information Not Collected or Collected Less Frequently.
These data are to be collected for this independent evaluation on a one-time basis. Not collecting these data would prevent CMHS from being able to gather the information it needs to be able to determine the extent to which the CMHS BG is fulfilling its legislative mandate.
7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2).
This data collection is consistent with the general information collection guidelines in 5 CFR 1320.5(d)(2).
8. Consultation Outside the Agency.
Federal Register announcement
The public notice soliciting comments on this information collection was published in the January 12, 2007 Federal Register (volume 72, number 81, page 1553 - 1555). Two pieces of public comment were received. The comments provided in the two letters are the following:
1) The evaluation team should review and abstract data from existing documents that States complete annually: the CMHS BG Application and Implementation Report.
Response: This process is an integral part of our data collection and analysis plan, but did not require OMB approval; therefore, it was not included in the Federal Register Notice. The Project Team has developed data abstraction forms to cull information from the CMHS BG Applications and Implementation Reports, and has begun the data abstraction process.
2) Interviews with planning council members may also be beneficial to the evaluation.
Response: With over 2000 Planning Council members nationally, it was difficult to develop an interview strategy that would be representative. Because most, if not all, of the State staff to be interviewed are housed in the State Mental Health Agency central offices, logistical arrangements and scheduling are streamlined. Many planning council members do not live in close proximity to the State Mental Health Agency requiring more intensive logistical arrangements and the expenditure of Federal resources to support their travel and time. The web-based survey for Planning Council Members was created to provide them with ample opportunity and method to provide as much qualitative information as would be contained in an interview format.
3) There is a concern that the collected information will be used to compare States and potentially penalize them for performance deficits.
Response: There is no intention to use the collected information to compare States in any way. All quantitative data will be aggregated and represented in that fashion in any project reports. Qualitative information will also be grouped with common themes identified. This data collection effort is aimed at better understanding the process by which SAMHSA and the States administer the program, and identifying any outcomes or longer-term impacts of the CMHS BG program. The purposes of the evaluation will be reiterated to all stakeholders when they are contacted to participate in the interviews and surveys.
b. Additional people consulted outside the agency
The agency’s contractor, contributed to the development of the survey instrument. The contractor’s address, phone number, and the staff persons involved are listed below:
Altarum
1200 18th Street NW Suite 700
Washington DC 20036
(202) 828-5100
Eric Gelman, M.B.A., M.A., Behavioral Health Director and Project Director
Scott L. Green, Ph.D., M.B.A., Senior Associate and Project Manager
Jessica McDuff, M.A., Senior Policy Associate
In addition, the Project Team consulted frequently with an Evaluation Advisory Workgroup (EAW) comprising a variety of individuals with evaluation skills and an in-depth knowledge of the CMHS BG program. The EAW members are the following:
Gregory Carlson, M.B.A. – Alabama Mental Health Planning Council Chair
John Hudgens, M.Ed. – Director of Community-based Services, Oklahoma Department of Mental Health and Substance Abuse Services
Debra Kupfer, M.M.H.S. – Former State Mental Health Commissioner, Colorado Department of Human Services
Gloria Logsdon, M.S. – Former CMHS BG Regional Reviewer and current CMHS BG Monitoring Site Visitor
Oscar Morgan, M.H.C.A. – Senior Consultant for Mental Health Policy and Program, Mental Health America (formerly the National Mental Health Association)
Daniel Powers, J.D. – Consumer Liaison, Nebraska Division of Behavioral Health Services
Sandra Spencer – Executive Director, Federation of Families for Children’s Mental Health
Dave Wanser, Ph.D. -- Deputy Commissioner for Behavioral and Community Health at the Texas Department of State Health Services
Michael Fitzpatrick, M.S.W. – Executive Director, National Alliance on Mental Illness
9. Remuneration of Respondents.
Respondents will not receive any payment.
10. Assurance of Confidentiality.
Personal information will not be collected. Respondents will be fully informed about the purpose of this study and the names of respondents will not be included in any reports from the study. Completed surveys will be maintained by contractor in a password-protected database. Information taken from interviews will be aggregated and presented as such in any reports developed for this project. Comments made through the interviews or surveys will not be attributable to specific individuals.
11. Questions of a Sensitive Nature.
This survey does not include questions of a sensitive or confidential nature.
12. Estimates of Annualized Hour Burden.
Data will be collected using two primary strategies: 1) on-site interviews with Federal and State staff, and 2) web-based surveys for Planning Council Members, Monitoring Site Visitors, and Regional Reviewers.
Estimate of interview burden. On-site interviews will be conducted with State Mental Health Agency staff who are knowledgeable about the CMHS BG program (as determined by the State Director). It is expected that interviews will typically include the State Director, State Mental Health Planner, and a State Mental Health Data Analyst. The interviews will be conducted in a group fashion and should last approximately 3.5 hours, based on pre-testing conducted by the contractor. Federal staff will also participate in on-site interviews that should last approximately one hour.
The estimated hourly wage of $37.09 is based on the mean hourly wage for Medical and Health Services Managers from the Bureau of Labor Statistics’ Occupational Employment Statistics for May 2005. The total estimated annualized cost to the Federal and State respondents is $23,719.05. This cost estimate was calculated based on the total respondent hour burdens and the estimated wage rate received from the Bureau of Labor Statistics.
Table 1 summarizes the estimate of the total annual time and cost burden to respondents resulting from participating in the interviews. Estimated burden was identified through pre-testing of the surveys and interview protocols conducted by the agency’s contractor.
Table 1: Estimated Reporting Burden of Interviews
Respondent |
Number of Respondents |
Average Hours per Interview |
Estimated Total Burden (hours) |
Estimated Mean Hourly Wage |
Estimated Total Cost |
State Mental Health Agency Commissioner |
59 |
3.5 |
206.5 |
$37.09 |
$7,659.09 |
State Planners |
59 |
3.5 |
206.5 |
$37.09 |
$7,659.09 |
State Data Analysts |
59 |
3.5 |
206.5 |
$37.09 |
$7,659.09 |
Federal CMHS Block Grant Staff |
20 |
1 |
20 |
$37.09 |
$741.80 |
Total Burden |
197 |
7 |
639.5 |
37.09 |
$23,719.05 |
Estimate of Web-based Survey burden.
Web-based surveys are an economical strategy for this data collection given that many of the stakeholders (e.g., Planning Council Members, Regional Reviewers) are geographically dispersed. Stakeholders will be able to access the survey through an internet connection and use a secure user interface to complete the survey. Based on system testing conducted by the contractor, it is estimated that completing the survey will take approximately one hour.
Given the potential variance in the occupations of stakeholders completing the web-based surveys, the estimated hourly wage of $18.21 is based on the mean hourly wage for all occupations from the Bureau of Labor Statistics’ Occupational Employment Statistics for May 2005. The total estimated annualized cost to the State respondents is $37,567.23. This cost estimate was calculated based on the total respondent hour burdens and the estimated wage rate received from the Bureau of Labor Statistics. Table 2 summarizes the estimate of the total annual time and cost burden to respondents resulting from completing the web-based survey.
Table 2: Estimated Reporting Burden of Web-based Surveys
Respondent |
Number of Respondents |
Average Hours per Interview |
Estimated Total Burden (hours) |
Hourly Mean Wage |
Estimated Total Cost |
Planning Council Members (including chairs) |
2000 |
1 |
2000 |
$18.21 |
$36,420.00 |
Regional Block Grant Reviewers |
35 |
1 |
35 |
$18.21 |
$637.35 |
Monitoring Site Visitors |
28 |
1 |
28 |
$18.21 |
$509.88 |
Total Burden |
2,063 |
1 |
2,063 |
$18.21 |
$37,567.23 |
13. Estimates of Annualized Cost Burden to Respondents.
There are no capital or start-up costs for this project.
14. Estimates of Annualized Cost to the Government.
The annualize cost to the Federal Government for this project is $807,865. This includes an estimate of $4,500 or 5% for a GS-14 employee’s time allocated as a project manager and $803,365 to the contractors for all other data collection activities.
15. Changes in Burden.
This is a new data collection.
16. Time Schedule, Publication and Analysis Plans.
a. Reports to be published
The contractor will write a summary report that includes a synthesis of findings from the conduct of this study, described in this clearance package. This report will include simple graphic displays of the key findings with detailed findings provided in a technical supplement.
The report will be structured as follows:
Executive Summary
Introduction
CMHS BG Purpose, History, and Requirements
Evaluation Purpose and framework
Purpose
Evaluation questions
Evaluation Advisory Workgroup
Report organization
Methods
Review of Source Documents
Development of data abstraction forms
Process of data abstraction
Development of Program Logic Model
Instrument Development and Revision
Surveys
Planning Council Chair
Planning Council members
Regional Reviewers
Monitoring Site Visitors
Interview protocols
State
Federal
State site visits
Web-based survey development and administration
Stakeholder interviews
Case Studies
Database Development
Data Analysis
Limitations
Results
CMHS BG Funding Distributions
Federal to State allocation process
State allocation of funds
Mechanism by which funds are allocated to subrecipients
How the allocation process meets CMHS BG legislative requirements
Influence of State laws and policies on funding distribution
Activities supported by the administrative set-aside
Strengths and areas for improvement in CMHS BG funding distributions
Application and Implementation Report Development and Review Processes
Development of application guidance for States
State application development and submission
State planning processes
Planning Council involvement
Application review and approval
Implementation report development
Implementation report review and approval
Types of Programs and Services Funded through the CMHS BG
Adult Programs and Services
Service modalities
Program types
Target populations served
Evidence-based practices
Unique and Innovative uses of CMHS BG funds
Strengths and areas for improvement in the types of programs and services funded through the CMHS BG
Children’s’ Programs and Services
Service modalities
Program types
Target populations served
Evidence-based practices
Unique and Innovative uses of CMHS BG funds
Strengths and areas for improvement in the types of programs and services funded through the CMHS BG
Program Development, Technical Assistance, and Training
Federally funded
State funded
Strengths and areas for improvement in program development
Program Monitoring by Federal Representatives
monitoring site visits
grants management
Strengths and areas for improvement with program monitoring
Evaluation of CMHS BG activities
Federally required data collection, analysis, and reporting
Uniform Reporting System data and NOMs
State performance indicators
Federal uses for data
State data collection from subrecipients
Data collection processes between States and subrecipients
Uses for data collected by States (non-Federal data)
Strengths and areas for improvement in evaluation
Programs Impacts and Contributions to Community-based Mental Health Systems of Care
b. Complex analytical techniques/Plan
The quantitative information derived from this survey will be entered into a database and analyzed using SAS, a statistical computing software program. Frequencies on all variables will be produced. Tests of significance may be used to analyze differences, such as between block grant allocation and numbers of services in States.
Qualitative information will be grouped by application section and content analyses will identify common themes among the information gathered. The analyses will also examine commonalities and differences in how the services are organized and implemented, and potential outcomes and impacts of the CMHS BG program on State mental health systems of care, and ultimately, the consumers who access services.
c. Time schedule
Table 3 reflects the schedule for each task in the design, data collection, and report compilation phases of the CMHS BG evaluation.
Table 3: Estimated time schedule for tasks
Task |
Estimated Start Date |
Estimated Completion Date |
Schedule and conduct interviews |
Upon OMB approval |
4 months post-OMB approval |
Launch web-based surveys |
1 month-post OMB approval |
3 months post-OMB approval |
Analyze data |
3 months post-OMB approval |
5 months post-OMB approval |
Draft Final Report |
6 months post-OMB approval |
8 months post-OMB approval |
17. Exemption for Display of Expiration Date.
The expiration date for OMB approval of the information collection will be displayed.
18. Exceptions to Certification for PRA Submissions.
No exceptions to the certification statement are requested. The certifications are included in this submission.
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Respondent Universes and Sampling Procedures
The respondents consist of the universe of the State Mental Health Agencies (SMHA) from each of 59 States and Territories. In addition, the respondents include the entire universe of Planning Council Members, Regional Reviewers, and Monitoring Site Visitors. Therefore, no complex sampling methods are required.
2. Procedures for Data Collection and Statistical Estimation
The data collection effort for this study will not employ any complex statistical methods in identifying the respondents. Federal staff involved in the administration of the CMHS BG program and the SMHA director from each of 59 States and Territories will be asked to participate via an introductory letter (see Attachment B), along with appropriate staff, in the interviews. Interviews will be conducted on-site and scheduled at the convenience of the SMHA staff.
Introduction letters will be sent to the remaining stakeholders (i.e., Planning Council Members, Regional Reviewers, Monitoring Site Visitors) asking for their participation, describing the web-based survey system, and providing instructions for accessing the system (see Attachment B). Access to the web-based surveys (via a URL hyperlink) will be sent via e-mail. For those stakeholders who do not have access to e-mail, hard copies of the survey can be mailed.
3. Maximizing Response Rates and Issues Related to Non-response.
Response rates and the on-site interviews. The contractor staff will work with each Federal staff person and SMHA to schedule the interviews at a time most convenient within our 4-month timeframe. This flexibility should enable us to have 100% response rate. The contractor will work with CMHS and the States to emphasize the importance of their participation in the evaluation.
Response rates and the web-based surveys. Given the relatively large number of stakeholders receiving the web-based surveys (n > 2000), maximizing response rates is extremely important. This will be handled in the following ways:
Respondents without access to e-mail will have hard copies of the survey sent to them with a stamped return envelope to facilitate return. Staff are also available to assist respondents by phone with completing the survey. Follow-up phone calls will be used to encourage respondents to complete and return the surveys.
The web-based survey system has an e-mail function that allows the contractor staff to send reminder e-mails to respondents if they have not accessed or completed the survey.
Given these strategies and that stakeholders have a vested interest in the CMHS BG program, we are anticipating that stakeholders will be cooperative with the evaluation. We anticipate a response rate of 75% for the web-based surveys.
4. Tests of Procedures.
The data collection instruments have been reviewed by CMHS and its contractor clarifying terminology and language and rewriting or eliminating questions that were unclear or unnecessary. Five States (State staff and Planning Council Members) also provided feedback on the instruments as part of the test of procedures.
5. Individuals Involved.
The data collection will be conducted by the contractor. The Project Director of the Independent Evaluation of the CMHS BG program is Eric P. Gelman, M.B.A., M.A.. The contact person for this survey is Scott L. Green, Ph.D., Senior Associate. Both can be reached at the address and number below:
Altarum
1200 18th Street NW Suite 700
Washington DC 20036
(202) 828-5100
• Eric Gelman, M.B.A., M.A.., Behavioral Health Director and Project Director
• Scott L. Green, Ph.D., Senior Associate and Project Manager
List of Attachments
Attachment A – Data Collection Instruments
A.1: Federal Interview Protocol
A.2: State Interview Protocol
A.3: State Planning Council Chairs
A.4: State Planning Council Members
A.5: CMHS BG Regional Reviewers
A.6: CMHS BG Monitoring Site Visitors
Attachment B – Introduction Letters
B.1: Federal Staff Introduction Letter
B.2: State Staff Introduction Letter
B.3: State Planning Council Members Introduction Letter (including Chairs)
B.4: CMHS BG Regional Reviewer Introduction Letter
B.5: CMHS BG Monitoring Site Visitor Introduction Letter
Attachment C – Web-based Survey Instructions
C.1: Web Survey Instructions
C.2: Reminder Email 1
C.3: Reminder Email 2
| File Type | application/msword |
| File Title | SURVEY OF SINGLE STATE AUTHORITIES REGARDING THE HIV SET-ASIDE OF THE SUBSTANCE ABUSE PREVENTION AND TREATMENT BLOCK GRANT |
| Author | blb |
| Last Modified By | SKING |
| File Modified | 2007-08-23 |
| File Created | 2007-08-03 |