THE CONSUMER LEVEL NATIONAL OUTCOME MEASURES (NOMs)
A. JUSTIFICATION
A.1 Circumstances of Information Collection
The Substance Abuse and Mental Health Services Administration’s (SAMHSA), Center for Mental Health Services (CMHS) is requesting from the Office of Management and Budget (OMB) approval for collection of outcome measures for mental health treatment services funded through the Center’s Programs of Regional and National Significance (PRNS) and the Children’s Mental Health Initiative (CMHI) budget lines. This approval will allow PRNS and CMHI grantees to report to CMHS on data for the Consumer National Outcome Measures (NOMs) identified for the mental health field. The authorization for the 2006 Consumer NOMs is Section 501(d)(4) of the Public Health Service Act (42 USC 290aa), which mandates the collection of statistical data on mental health programs and on the persons who receive care from them.
Approval of this information collection will allow SAMHSA to continue to meet Government Performance and Results Act of 1993 (GPRA) reporting requirements that quantify the effects and accomplishments of its programs, which are consistent with OMB guidance. In order to carry out section 1105(a)(29) of GPRA, SAMHSA is required to prepare a performance plan for its major programs of activity. This plan must:
a) Establish performance goals to define the level of performance to be achieved by a program activity;
b) Express such goals in an objective, quantifiable, and measurable form;
c) Briefly describe the operational processes, skills and technology, and the human, capital, information, or other resources required to meet the performance goals;
d) Establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity;
e) Provide a basis for comparing actual program results with the established performance goals; and
f) Describe the means to be used to verify and validate measured values.
This data collection will provide CMHS to use the Consumer NOMs as the GPRA measures for mental health treatment services funded through the PRNS and CMHI budget lines. This would result in a consistent set of GPRA measures across the various grant programs funded under these budget lines.
The information collection also will allow SAMHSA to respond to the Office of Management and Budget Program Assessment Rating Tool (PART) evaluations. This information collection will allow CMHS to be consistent with the specific performance domains that SAMHSA is implementing, known as the National Outcomes Measures (NOMs), to assess the accountability and performance of its grant programs.
SAMHSA’s legislative mandate is to increase access to high quality substance abuse and mental health prevention and treatment services and to improve outcomes. Its mission is to improve the quality and availability of treatment and prevention services for substance abuse and mental illness. To support this mission, the Agency’s overarching goals are:
1) Accountability—Establish systems to ensure program performance measurement and accountability
2) Capacity—Build, maintain, and enhance mental health and substance abuse infrastructure and capacity
3) Effectiveness—Enable all communities and providers to deliver effective services
Each of these key goals complements SAMHSA’s legislative mandate. All of SAMHSA’s programs and activities are geared toward the achievement of these goals and GPRA performance monitoring is a collaborative and cooperative aspect of this process.
SAMHSA will strive to coordinate the development of these goals with other ongoing performance measurement development activities. This information collection is needed to provide objective data to demonstrate SAMHSA’s monitoring and achievement of its mission and goals.
A.2 Purpose and Use of Information
This proposed data activity is to provide a level of consistency with the NOMs domain in SAMHSA-wide data collection and reporting of performance measures across all of its Centers and programs. This particular activity will promote the use of consistent measures among CMHS programs, grantees and contractors funded through the Programs of Regional and National Significance (PRNS) and the Children’s Mental Health Initiative (CMHI) budget lines. The Consumer NOMs recommended by CMHS are a result of extensive examination and recommendations, by panels of staff, experts, and grantees. Wherever feasible, the proposed measures are consistent with or build upon previous data development efforts within CMHS.
Individuals at three different levels will use the information: the SAMHSA administrator and staff, the Center administrators and government project officers, and grantees:
SAMHSA Level—The information will be used to inform the administration on the performance of the programs funded through the Agency. Performance will be based on the goals of the grant program and will include the new Consumer NOMs. The intent is that the information will serve as the basis of the annual GPRA report to Congress contained in the Justifications of Budget Estimates.
Center Level—In addition to providing information on the performance of the various programs, the information can be used to monitor and manage individual grant projects within each program. The information can be used to identify strengths and weaknesses and provide an informed basis for providing technical assistance and other support to grantees, informing continuation funding decisions, and identifying potential subjects for further evaluation.
Grantee Level—In addition to monitoring performance outcomes, the grantee staff can use the information to improve the quality of treatment services that are provided to consumers within their projects.
To fulfill GPRA requirements SAMHSA develops a report for each fiscal year that includes results of performance monitoring for the three preceding fiscal years. SAMHSA and CMHS intend to compare consumer data collected at baseline with periodic reassessments. These outcomes will be used as the measure of performance. The additional information collected through this process will allow SAMHSA to report on the results of these performance outcomes as well as be consistent with the specific performance domains that SAMHSA is implementing to assess the accountability and performance of its discretionary and formula grant programs.
To facilitate SAMHSA-wide reporting, the agency has identified ten domains of particular interest for accountability and performance monitoring. These domains are:
Access/Capacity
Functioning
Stability in Housing
Education and Employment
Crime and Criminal Justice
Perception of Care
Social Connectedness
Retention
Cost-Effectiveness
Evidence-Based Practices
CMHS has identified consumer-level measures for eight of ten domain elements, which are the focus of this request. The measurement strategy for cost-effectiveness and evidence-based practices domains are under development and will be forwarded for OMB review in a separate submission. Specific measures in each of the remaining eight domains have been identified. Three instruments were developed; one for programs serving adults and the other two for programs serving children. (See Attachment 1 Adult Consumer NOMs Data Collection Tool, Attachment 2 Child Consumer NOMs Data Collection Tool Caregiver Respondent, and Attachment 3, Child Consumer NOMs Data Collection Tool Child or Adolescent Respondent.) The two children’s versions contain identical items. The only distinction is that one is designed for the caregiver to serve as the respondent while the other, intended for older children allows for the child to serve as the respondent. Individual providers have the discretion to administer the survey to either the caregiver or the youth. However, a consistent approach to data collection will be used for all respondents; NOMs data will be collected at baseline with a periodic reassessment being conducted as long as the consumer remains in treatment. The use of standardized domains and data collection approaches will enhance aggregate data development and reporting. The following table summarizes the number of items for each domain:
Domain |
Number of Items: Adult |
Number of Items: Caregiver and Child/Adolescent |
Access/Capacity |
4 |
4 |
Functioning |
8 |
6 |
Stability in Housing |
1 |
2 |
Education and Employment |
4 |
3 |
Crime and Criminal Justice |
1 |
1 |
Perception of Care |
14 |
13 |
Social Connectedness |
4 |
4 |
Retention1 |
0 |
0 |
Total Number |
36 |
33 |
In addition to questions asked of consumers related to the NOMs domains, programs will be required to report information on consumer demographics at baseline and abstract information from consumer records on the services received.
A.3 Uses of Information Technology
Information technology will be used to reduce program respondent burden. Modifications will be made to the existing Service Accountability Improvement System (SAIS)—a web-based data entry and reporting system originally designed for Center for Substance Abuse Treatment (CSAT) GPRA data collection — to support web-based data collection efforts for CMHS. The modified CMHS electronic system, referred to as TRAC (Transformation Accountability) System, will provide a data repository service that accommodate the specific NOMs consumer measures. This web-based repository service will include methods for receiving the data, data quality checks, storage, and data presentation in reports by individual performance measure or grouped with other performance measures.
This web-based system is intended to allow for easy data entry and access to reports for grantees that are required to submit the NOMs data for consumers. Levels of access will be defined for users based on their authority and responsibilities regarding the data and reports. Access to the data and reports will be limited to those individuals with a username and password.
Electronic submission of the data promotes enhanced data quality. With built in data quality checks, and easy access to data outputs and reports, users of the data can feel confident about the quality of the output. The electronic submission also promotes immediate access to the dataset. Once the data are put into the web-based system, it will be available for access, review, and reporting by all those with access to the system from Center staff to the grantee staff. All data entry screens will include the OMB approval information. (See Attachment 4: TRAC Screen Shots).
A.4 Efforts to Identify Duplication
The items collected are necessary in order to assess grantee performance. Currently, CMHS does not have standard reporting for grant programs that provide treatment services to consumers, so this information is not available elsewhere. However, individual grant programs are independently collecting various types of measures.
Individual CMHS programs currently collect and report some type of GPRA data. As a result, a program-level review of current measures and methods of collection was conducted to identify duplication of the proposed data collection effort. With the goal of creating standardized methods and measures for monitoring grantee performance across the Center, existing measures were considered for use where appropriate. However, modification of current measures was precipitated in some cases given the variation across programs. The proposed items were reviewed and approved by the Government Project Officers, Branch Chiefs, and CMHS Senior Leadership as meeting the performance monitoring and management needs of individual programs and the Center.
A.5 Involvement of Small Entities
Individual grantees vary from small entities through large provider organizations. Every effort has been made to reduce the number of data items collected from grantees to the least number required to accomplish the objectives of the effort and to meet GPRA reporting requirements and therefore, there is no significant impact involving small entities.
A.6 Consequences if Information Collected Less Frequently
Mental health programs typically collect data at admission and then conduct periodic reassessments of consumers while the individual remains in treatment. When feasible, mental health providers also conduct an assessment when the consumer is discharged. The proposed data collected for the CMHS NOMs parallels this model. Standard clinical practice guided the decision to require that grantees that tend to provide shorter term treatment collect the data on a consumer every three months. This includes grantees funded through the National Child Traumatic Stress Initiative and the Meeting the Needs of Elderly Americans and HIV/AIDS Minority Mental Health Services Programs. The remaining grantees, funded through CMHI and Criminal Justice Jail Diversion program will conduct the periodic reassessments every six months.
The baseline data collection point is critical for measuring changes. Extending the interval for the periodic reassessment beyond the requested intervals could lead to loss of contact with consumers/participants, significantly diminishing the response rates and lowering the value of the data for GPRA use by losing measurement of intermediate effects.
A.7 Consistency with the Guidelines in 5 CFR 1320.5(d)(2)
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
A.8 Consultation Outside the Agency
CMHS consulted both external and internal stakeholders in developing the proposed measures and data collection methodology. CMHS obtained feedback and consultation regarding the availability of data, methods and frequency of collection, and the appropriateness of data elements.
Development of the measures involved extensive consultation with staff within CMHS and SAMHSA. In many instances, CMHS staff also sought feedback from their grantees to inform their thinking. Staff involved in this process included:
Emergency Mental Health Seth Hassett, M.S.W., Mikisha Nation, M.Ph.
and Traumatic Stress
Services Branch
Prevention Initiatives and Susan Keys, Ph.D., Barbara Silver, Ph.D.
Priority Programs Development
Branch -Medical Affairs
Homeless Programs Larry Rickards, Ph.D., Pamela Fischer, Ph.D.
Child, Adolescent & Gary Blau, Ph.D., Sylvia Fisher, Ph.D.
Family Services
Community Support Neal Brown, M.P.A., Crystal Blyler, Ph.D., David
Programs Branch Morissette, D.S.W., Betsy McDonel-Herr, Ph.D.
State Planning and Systems Marie Danforth, M.S.W., Karen Armstrong,
Development Branch M.S.W., J.D., Richard DiGeronimo, Paul Wohlford
Survey and Analysis Branch Olinda Gonzalez
Internal Consultations (SAMHSA):
OPPB/SAMHSA Coordinator Suzanne Fialkoff
Office of Applied Studies Sarah Duffy, Ph.D.
Office of the Administrator Sue Becker
The notice required by 5 CFR 1520.8(d) was published in the Federal Register on June 9, 2006 (vol. 71, no.111, page 333476). Six sets of comments were received in response to this notice. (See Attachment 4 for a copy of comments received.)
Comments were received from:
Mary Armstrong
Shannon Crossbear
Co-Chairs
Outcomes Roundtable for Children and Families
Denise M. Cole
Director of Quality Improvement for Behavioral Health Services
111 Elwyn Road
Elwyn, PA 19063
Barbara E. Footer, M.S.
Program Manager for Behavioral Health
HSD/Medical Assistance Division
2025 So. Pacheco Street
Santa Fe, NM 87504
Judy Hall, Ph.D., Chair
Mary E. Smith, Ph.D., Immediate Past Chair
Mental Health Statistics Improvement Program
Michael F. Hogan, Ph.D.
Director
Ohio Department of Mental Health
30 East Broad Street
Columbus, OH 43215
David Johnson, ACSW
Chief Operating Officer
University Psychiatric Centers
2751 E. Jefferson
Detroit, MI 48207
Many of the comments commended SAMHSA and CMHS for their efforts to develop a common set of performance measures such as the NOMs that could be used across different grant programs within the Center. At the same time some concerns were expressed about the general thrust of implementing NOMs.
The timetable for implementation was seen as too aggressive and 2-3 years should be allocated for CMHS to implement this data collection effort. Some of the comments suggested that a program of research be undertaken to develop the measures to be used.
Although SAMHSA and CMHS recognize the challenges in implementing a standardized reporting system across the five CMHS programs that fund treatment services for consumers, the demand for program accountability requires CMHS to move forward expeditiously. The most recent OMB Program Assessment Rating Tool (PART) review of the CMHS PRNS budget line found that results were not demonstrated. In its improvement plan to address the deficiencies noted, CMHS stated:
We are taking the following actions to improve the performance of the program:
Implementing automated web-based performance system including development and implementation of common performance measures on which all grantees must report.
Developing measures of program efficiency and the program's impact on increasing functioning of mental health patients.
Focusing program resources on areas that most directly contribute to the service mission of the program.
The proposed data collection is an integral part of the actions taken by CMHS to address the concerns noted in the PART review and the Center must move ahead with this effort with the greatest speed possible. CMHS will make necessary refinements of the proposed measures as experience is gained with them.
One commenter raised a concern about the use of a standard set of measures across a diverse set of programs that serve varied consumer types.
CMHS agrees that there are major challenges in developing a common set of measures that could be meaningful across the various grantees. However, the Center must have measures in place that cut across grantees and programs to respond to accountability requirements such as PART. CMHS program staff met many times to discuss possible measures and to reach consensus on this set of standard measures that could be used across the varied programs.
Another set of concerns related to costs associated with the implementation of the NOMs. One comment provided specific costs associated with the implementation of an outcomes monitoring system in their jurisdiction, with one of the major expenditures relating to the redesign and development of their data system. A related comment suggested that technology be used to help minimize the costs of the data collection.
In developing the proposed NOMs measures, CMHS continuously considered the potential burden on grantees who must submit the data. A guiding principle was to keep the actual number of questions or measures at the minimum necessary to address the SAMHSA NOMs domains. CMHS will also make use of information technology (IT) by developing a web-based data entry system and reporting systems for use by all grantees. This platform will serve as the repository for the data so grantees will not be required to modify their existing IT systems to comply with this requirement.
A related cost concern was the expense for grantees to locate and interview consumers, particularly those who are not longer involved with their programs.
Currently, grantees will only be required to provide Periodic Reassessment data on those consumers who are participating in their program. The CMHS NOMs at this time do not require grantees to track consumers who are no longer in the program.
Concerns were expressed about the low proportion of consumers that would provide outcomes based on data from the Periodic Reassessment.
The plan at the time of the publication of the Federal Register notice was to require grantees to complete the Periodic Reassessment at six months intervals. In response to the comments received, CMHS has reconsidered this approach, recognizing the limitations with only having outcome data based on the Periodic Reassessment on only 35% of the consumers, the proportion estimated to remain in treatment for at least six months.
Discussions were held with CMHS representatives of the various grant programs to identify a solution that would provide outcome data on a larger proportion of clients but not require grantees to track consumers after they left the program. The compromise position was to collect Periodic Reassessment data at three months for three of the five CMHS grant programs required to submit NOMs data. The two other programs, CMHI and Jail Diversion tend to serve consumers for a longer period of time than other programs so a decision was made to maintain the six month interval for the Periodic Reassessment. The Periodic Reassessment would be recurring and collected at these intervals as long as the consumer remains in treatment. This change will result in CMHS having outcome data from the Periodic Reassessment on a much larger proportion of consumers and still avoid the burden associated with locating consumers no longer involved with the grantee.
We believe this change will greatly strengthen the value of the NOMs data for mental health consumers to SAMHSA and CMHS. However, it does increase the burden on grantees by increasing the number of Periodic Assessments to be conducted. The burden table in this supporting statement reflects this increase.
Several of the comments focused on specific questions proposed. Many of these comments were related to the Mental Health Statistics Improvement Program (MHSIP) Consumer Survey, the Youth Services Survey/Family (YSS/F), and the Youth Services Survey which served as the starting points for many of the questions on the adult and youth data collection instruments respectively. Comments included:
One recommendation was that CMHS use all of the items on these surveys.
The decision was made to use all of the items from these surveys with the exception of the six questions from the adult survey and four questions from the child and family instruments that apply to access. The NOMs access domain will be measured using counts of number of consumers served and their demographic characteristics.
A comment was received suggesting that the phrase “as a result of treatment I received” be retained on the questions in the functioning domain on both the adult and children’s version of the NOMs survey to maintain comparability with the MHSIP, YSS/F and YSS.
The functioning questions will be asked at both Baseline and at the Periodic Reassessment to allow CMHS to measure change in this domain. The phrase in question is not appropriate at admission and the desire was to use the same wording across interviews.
There is a need for precise definitions of terms to ensure consistent data collection across the programs.
CMHS concurs with the comment. There are plans to develop training and support documentation to aid the interviewer in the specific parameters pertaining to administering the tool. Operational definitions will be set forth in the Question-by-Question specification.
One education question jointly considers school enrollment and participation in a job training program. The suggestion was to distinguish between the two types of activities.
The intent of this question was to determine whether the consumer was actively engaged in activities designed to better their employment opportunities. CMHS does not believe distinguishing between the two types of activities is necessary in this situation.
The response categories for the Employment question should not include “sheltered workshop or clubhouse”.
The tool was revised to exclude these items under the response “Supported Employment.”
Clarification was requested regarding the use of disability in the employment question.
The intent of the question is not to determine whether someone is disabled, but whether their disability is the reason he/she is unemployed. A person could be disabled and not looking for work or retired, and thus would not need to indicate their disability (the response items include “unemployed, not looking for work; unemployed, retired”, etc). This will be addressed more explicitly in the Question-by-Question guide.
Several of the proposed questions ask about the previous 30-day period (e.g., number of arrests.) A longer period of time should be used. This would allow for more stable results and capture information across the entire time span since the prior interview.
The 30-day window was selected in response to concerns regarding recall bias and difficulties in anchoring time with a longer interval. This window does provide a “snapshot” of a consumer’s current status.
One comment recommended: “Use the following introductory statement for the social connectedness section: Please answer for relationships with persons other than your mental health provider(s).”
The survey was modified to include this introductory statement.
One comment asked for an operational definition of “Unsatisfactory/Satisfactory” in the question covering discharge status.
This response category was removed from both the adult and child instruments.
A.9 Payment to Respondents
No monetary payment will be made to the mental health programs or to the consumers participating in the survey.
A.10 Assurance of Confidentiality
This CMHS data collection process involves gathering confidential information. Program respondents will be expected to meet the requirements of the HIPAA and its associated Privacy Rule that sets the standards for the use and disclosure of an individual’s health/mental health information. Since the data reported for each consumer/patient will be provided to the CMHS contractor only by number and not by name, the data cannot be directly linked to a specific person. The grantee providing the data will maintain the link between the identifier and the name of the consumer. The CMHS contractor will not have access to existing consumer/patient clinical records, which are under the control of the respondent programs. Neither the CMHS contractor nor CMHS can link individual consumers/ patients to the data reported by the respondent programs.
A.11 Questions of a Sensitive Nature
SAMHSA’s mission is to improve the quality and availability of prevention, early intervention, treatment, and rehabilitation services for substance abuse and mental illnesses, including co-occurring disorders, in order to improve health and reduce illness, death, disability, and cost to society. In carrying out this mission it is necessary for service providers to collect sensitive items such as criminal justice involvement as well as issues of mental health. The data that will be submitted by each grantee will be based in large part on data that most of the programs are already routinely collecting. This primarily includes data on consumer demographics, mental health condition/illness and treatment history, services received, and consumer outcomes. These issues are essential to the service/treatment context. Grant projects use informed consent forms as required and as viewed appropriate by their individual organizations. They use the appropriate forms for minor/adolescent participants requiring parental approval.
Following is the estimated annual response burden for this effort.
Data Collection Activity/Number Of Annual Participants |
Data Collections Per Client/ Participant |
Hours Per Data Collection |
Total Hour Burden |
Hourly Rate - Minimum Wage |
Total Hour Cost -Client/ Participants |
|||||||||
Baseline |
|
|
|
|
|
|||||||||
17,555 |
1 |
0.333 |
5,852 |
$15 |
$87,775 |
|||||||||
Periodic Assessment |
|
|
|
|
|
|||||||||
7,015 (3-month ) 6,532 (6-month) |
1 1 |
0.333 0.333 |
2,338 2,178 |
$15 $15 |
$35,075 $32,662 |
|||||||||
Discharge Interviews1 |
|
|
|
|
|
|||||||||
4,409
|
1 |
0.333 |
1,470 |
$15 |
$22,045
|
|||||||||
Record Abstraction2 |
|
|
|
|
|
|||||||||
7,015 (3-month ) 6,532 (6-month) 11,023 (Discharge) |
|
0.1 0.1 0.1 |
|
$15 $15 $15 |
|
|||||||||
TOTAL |
|
|
14,294 |
|
$214,427 |
Note: This is the maximum additional burden if all consumers/participants complete the baseline and periodic reassessment interviews.
Based on an estimate that it will be possible to conduct discharge interviews on 40% of those who leave the program.
Record abstraction will be conducted on 100% of those discharged.
A.13 Estimates of Annualized Cost Burden to Respondents
There will be no capital, start-up, operation, maintenance, nor purchase costs incurred by the mental health programs participating in this CMHS data collection, or by consumers receiving CMHS-funded treatment services.
A.14 Estimates of Annualized Cost to the Government
The total contract award to cover all aspects of the design of the study, sampling design, data collection, and development of the data files, data tapes, and technical documentation is $1,759,582 over a 36-month period. Thus, the annualized contract cost is $586,527.
Additional costs will be incurred indirectly by the government in personnel costs of staff involved in oversight of the survey. It is estimated that two CMHS employee will each be involved for approximately 44% of their time. Costs of CMHS staff time will approximate $53,541 annually.
The estimated annualized total cost to the government will be $640,068.
A.15 Changes in Burden
This is a new information collection.
A.16 Time Schedule, Publication and Analysis Plans
Data for the annual GPRA plan/report are needed by SAMHSA on an ongoing basis. Data collection will commence with approval from OMB. Data are provided by CMHS for the most recently completed calendar year to SAMHSA each May in order to assure analysis in time for the annual GPRA report. The annual GPRA report must be submitted to the Department of Health and Human Services and to OMB by September and is included in the President's Annual Budget Request, which is released to the public February 1st. Data may be refined and added to the final Presidential Budget Request after the Department submits its initial GPRA report.
Publication Plan
Data will be available to CMHS staff and grantees through a series of reports available through the web-based TRAC system. Roles will determine user access. Individual grantees will only be allowed detailed access to data from their grant. They will also have access to summary information across all grantees in their program. CMHS staff access will be determined by their span of responsibility.
Data Analysis Plans
The web-based reports on the TRAC system will include information on the number of consumers served, their demographic characteristics, baseline status, and change scores for the various domains. The data items collected will be analyzed and presented in GPRA reports using basic descriptive statistics. On the principle outcome items (i.e., the 8 NOMs domains covered), a comparison of client status after treatment with baseline data will be used to assess any change in status. The web-based reports will also allow users to create basic cross tabulations of the data.
Data will be used to report to Congress regarding the GPRA as specified in the SAMHSA Annual Justifications of Budget Estimates. They will also allow CMHS staff to examine performance longitudinally, by program, or individual grantee.
In addition to the reports on the TRAC web site, data will be utilized for specialized analyses as needs emerge. Individual grantees will be able to download their own data in into an Excel spreadsheet for further manipulation or to transfer to a statistical package.
The expectation is that over time the results will be examined for subpopulations of interest within individual activities (e.g., by age or by gender) or in response to emerging policy issues. With these analyses the data would be exported to a statistical package such as SAS for more elaborate analytic work.
A.17 Display of Expiration Date
The expiration date for OMB approval will be displayed on all data collection instruments for this survey (See Attachment 4: TRAC Screen Shots.)
A.18 Exceptions to Certification Statement
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Respondent Universe and Sampling Methods
Selection of Mental Health Programs
All grantees funded through Programs of Regional and National Significance (PRNS) and the Children’s Mental Health Initiative (CMHI) within its discretionary program portfolio that provide direct clinical services to mental health consumers will be required to submit the NOMs data. This includes five grant programs (with the number of active grantees serving consumers in FY 2007 listed in parentheses):
Children’s Mental Health Initiative (54)
National Child Traumatic Stress Initiative (41)
Meeting the Needs of Elderly Americans (11)
Criminal Justice Jail Diversion (12)
HIV/AIDS Minority Mental Health Services (16)
B.1.2 Selection of Consumers/Patients—
SAMHSA data policy recognizes the value of sampling when it can be appropriately done. CMHS has agreed to allow sampling with consumer measures under the following conditions:
Programs must create a sampling plan including a detailed analysis plan that is consistent with the purpose of the ongoing collection of GPRA measures (i.e., program management and grantee oversight) and also reflects reasonably foreseeable data demands CMHS will face (e.g., information by key demographic characteristics.)
The sampling plan must yield annual estimates for the NOMs measures at the grantee level.
If an individual grantee treats less than 100 consumers per year, then NOMs data must be collected from all consumers.
Methods and resources must be in place for independent quality control and ongoing monitoring to ensure that the sampling plan is implemented as designed.
Programs must deliver sample weights associated with each observation.
One grant program, the Children’s Mental Health Initiative (CMHI) will sample consumers at nine of its largest grantees. The other four programs will collect NOMs data from all consumers served.
CMHI will use a simple random sample at these nine grantee sites, selecting every nth consumer, with the sampling rate set to yield 100 consumers per year from those grantees.
B.2 Information Data Collection Procedures
B.2.1 Data Collection Procedures and Instruments
Information data collection procedures will be the responsibility of individual grantees and may vary by type of program. Intake/baseline information is obtained by intake workers and/or counselors. For consumers still in treatment 6 months later, the information will most likely be collected by the counselor as part of the case review.
Some programs collect their client/participant information using paper and pencil methods. This project will not interfere with ongoing program operations. Programs will submit their data electronically via a web-based data entry process or upload process. The data for those clients/participants with both baseline and periodic reassessment data are matched using a unique encrypted consumer identifier developed by the grantee. Grantees will be clearly instructed not to use identifying information (i.e., social security number) as the patient identifier.
Required data collection points are:
BASELINE: For consumers who have not previously been seen by the grantee, baseline data will be collected at admission. For consumers already enrolled in the program, baseline data should be collected during the first contact with the grantee during the CMHS grant period. The timing of any subsequent data collection point(s) will be anchored to the baseline point.
PERIODIC ASSESSMENT: CMHS requires NOMs data collection every six months while the consumer is receiving CMHS-funded services. Ongoing periodic status review is viewed as consistent with good clinical practice. Three of the programs will be conducting periodic reassessments at 3-month intervals. CMHI and Jail Diversion will use a six month interval.
DISCHARGE: Grantees must provide information on the type of discharge on all consumers who are discharged. When the discharge is a planned event, the consumer will also be asked the questions on the CMHS NOMs data collection tool. The one exception to this requirement is when a consumer had responded to these same questions within the past 30 days as part of a Periodic Reassessment.
B.3 Methods to Maximize Response Rates
Each grantee will have established its own procedures to collect baseline, periodic reassessment, and discharge data as part of the original protocol. For newly admitted consumers, baseline data collection would typically occur at the time of intake. All other data collection would occur as part of the normal course of treatment, most likely by the primary counselor or clinician assigned to the consumer. As noted, the timing of the periodic reassessment was chosen to coincide with normal clinical practice. Consumers/participants are typically quite cooperative with grantee staff because of the relationship established during treatment. Since all participating grant programs will collect data at initial intake, considerable options also exist for non-respondent analysis and associated adjustments to the data such as weighting.
A relevant feature of the web-based system (TRAC) that will support the Consumer NOMs data collection activities is that it will automatically generate notices of when periodic assessment interviews are due for each consumer/participant. Training on this and other features of the TRAC system will be provided at national grantee meetings. In addition to these training sessions, experts as well as selected grantees will be identified and asked to make presentations at national grantee meetings on the importance of quality and complete data collection, as well as TRAC system features to help facilitate consistency on consumer assessments at the appropriate intervals. Since these sessions are well attended by grantees, it is anticipated that these strategies will help to improve completion rates.
B.4 Tests of Procedures
All of the data elements in the proposed data sets have been taken from established data collection instruments that have a history of use in the mental health field and have already been tested for validity and reliability, (i.e., MHSIP, YSS-F, and YSS). In addition, for the domains that are not specific to mental health, CMHS has taken questions currently used by CSAT (OMB No. 0930-0208) that were drawn from widely used instruments and have been used for several years. These include three consumer-level domains (Employment/Education, Crime and Criminal Justice, and Stability in Housing) and one system-level domain (Access/Capacity), which depends on common demographics collected on consumers. The content of these questions was appropriate for use, but additional value options were defined to reflect issues specific to the populations served by CMHS. The benefits of using these measures include a history of use in monitoring the performance of CSAT grantees, the ability to conduct cross-Center comparisons, and use of measures previously approved by OMB.
B.5 Statistical Consultants
CMHS has contracted with Westat to provide support for the development and ongoing operational support for the NOMs effort, including statistical and analytic issues and the development of a web-based reporting the system. The Westat Project Director for this effort is:
Bill Luckey, Ph.D.
Vice
President & Associate Director
Substance Abuse Research
Group
Westat
1650 Research Blvd.
Rockville, MD
301-610-4861
Individuals Consulted on Statistical Aspects of the Design
Bill Luckey, Ph.D. (301-610-4861) and Jessica Taylor, Ph.D. (240-314-5852) are the individuals responsible for statistical consultation for this data collection.
Individuals Collecting and/or Analyzing Data
The following individuals are the CMHS personnel responsible for receiving and approving contract deliverables.
Diane Abbate
Project
Officer
DHHS/SAMHSA
Center for Mental Health
Services
Office of he Director
1 Choke Cherry
Road
Rockville, MD 20857
240-276-1824
Mark
Jacobsen, Ph.D.
Alternate Project Officer
Senior Program
Management Specialist
DHHS/SAMHSA
Center for Mental Health
Services
Office of Policy, Analysis, and Coordination
1 Choke Cherry Road
Rockville, MD 20857
240-276-1826
List of Attachments:
Attachment 1 Adult NOMs Data Collection Tool
Attachment 2 Child NOMs Data Collection Tool, Caregiver Respondent Version
Attachment 3 Child NOMs Data Collection Tool, Child or Adolescent Respondent
Version
Attachment 4 Web-based Data Entry Screen Shots
Attachment 5 Comments Received in Response to the Federal Register Notice
Attachment 1
Adult NOMs Data Collection Tool
Attachment 2
Child NOMs Data Collection Tool, Caregiver Respondent Version
Attachment 3
Child NOMs Data Collection Tool, Child or Adolescent Respondent Version
Attachment 4
Web-based Data Entry Screen Shots
Attachment 5
Comments Received in Response to the Federal Register Notice
1 [1] Retention is defined as retention in the community. The indicator is based on use of psychiatric inpatient services, which is based on a measure from the Stability in Housing Domain.
| File Type | application/msword |
| File Title | THE 2005 CLIENT/PATIENT SAMPLE SURVEY |
| Author | Dennis Sayre |
| Last Modified By | SKING |
| File Modified | 2006-12-15 |
| File Created | 2006-12-13 |