CMHS MHCPE Supporting Statement Final 3 17 2011(2)

PARTICIPANT FEEDBACK FORMS FOR THE

MENTAL HEALTH CARE PROVIDER EDUCATION

IN THE HIV/AIDS (MHCPE) PROGRAM

SUPPORTING STATEMENT

A. JUSTIFICATION

1. Circumstances of Information Collection 

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS) is requesting from the Office of Management and Budget (OMB) approval of an extension for the use of standardized forms to collect systematic feedback from trainees participating in the Mental Health Care Provider Education in HIV/AIDS (MHCPE) Program. CMHS supports education for mental health providers through its HIV/AIDS education programs. The feedback forms and program assessment design for this program are used by education site staff in the current CMHS MHCPE Program and are approved under OMB No. 0930-0195, which expires January 31, 2011. No revisions are being proposed for the Session Report Form (completed by education site staff). There are four post-session forms (Attachment A), and no revisions are being proposed. CMHS is authorized to collect the data under 42 USC 290aa (Section 501(d) (4)) of the Public Health Service Act.

The overall goal of the education program is to help create a cadre of traditional and non-traditional mental health service providers who possess and utilize state-of-the-art information on the psychological and neuropsychological sequelae of HIV/AIDS, and to enhance the nation’s ability to have an impact on the HIV/AIDS epidemic. CMHS has used the participant feedback forms and over-all assessment design for over 10 years in its MHCPE Program. CMHS has used the multi-site assessment data to verify the integrity and efficacy of these organizations’ efforts to educate mental health workers, and thereby enhance the quality of services available to HIV-affected individuals. This information allows CMHS to continue to assess its success in creating a cadre of mental health service providers for HIV/AIDS-affected populations.

In August 2008 the Centers for Disease Control and Prevention (CDC) published the first national HIV incidence (new infection) estimates for the United States using new technology and methodology that more directly measure the number of new infections. Based on data from 2006, CDC reported an estimated 56,300 new HIV infections occurred, substantially higher than the previous estimate of 40,000 annual new infections, and also confirmed that gay and bisexual men of all races, African Americans, and Hispanics/Latinos were most heavily affected by HIV. It is estimated that in the United States between 950,000 and 1.2 million people are currently infected with HIV, and as of 2005, over 984,155 reported cases of AIDS (CDC, 2010). In addition, people of color living with HIV/AIDS continue to become critically ill and/or die at distressing rates despite widespread availability of highly effective HIV/AIDS medical treatments in the U.S. (Kaiser Family Foundation Fact Sheet, October 2008).

There is a continued growth in the need for mental health treatment for HIV affected individuals. Untreated and undiagnosed neuropsychiatric complications related to HIV and AIDS often lead to more serious problems, such as non-adherence with the treatment regimen, impaired quality of life, and increased morbidity and mortality. Individuals affected by HIV/AIDS confront critical life altering decisions in view of changing options for medical treatment particularly protease inhibitors and Highly Active Anti-Retroviral Therapy. Given the effects of HIV disease itself, coupled with effects of medication used to treat it, continuing education and relatively frequent updates for mental health services providers about developments in the treatment and psychological aspects of HIV care are crucial. The mental health practitioner’s role has become increasingly significant as the psychosocial and cultural issues surrounding the treatment of HIV/AIDS continue to grow in complexity. Mental health practitioners more than ever need to acquire training specific to the mental health needs of HIV-affected individuals across a wide variety of populations.

The MHCPE Program currently provides funding to three mental health professional associations: the American Psychological Association (APA), the American Psychiatric Institute for Research and Education (APIRE), and the National Association of Social Workers (NASW), and potentially for additional education site grantees, thus, the estimates of burden/cost are based on 10 sites. These trainers help to train and educate mental health professionals in their respective disciplines; taken together, the cadre of mental health professionals trained by these associations comprise a significant proportion of mental health providers that serve the HIV/AIDS affected population in our nation.

The theoretical and practical foundation for this round of funding comes from 18 years of prior CMHS experience through its HIV/AIDS education programs. The CMHS MHCPE Program was designed to develop model approaches to educate mental health care providers in the neuropsychiatric, ethical and psychosocial aspects of HIV/AIDS. For over 10 years the MHCPE Program has funded education for mental health providers, and has conducted a multi-site assessment of the program. Over this period the MHCPE Program conducted more than 2,278 training sessions, and collected feedback regarding, for example, satisfaction with training and knowledge gained through training from over 36,300 participants. This represents an over-all response rate at almost or over 80%, for two organizations and slightly lower for the third. The lower rate for APIRE pertains to the reduced amount of time for each training session including ‘drop-in’ training such as grand-rounds in a hospital setting during the medical work-day. CMHS is able to effectively assess its MHCPE Program through this process over-all. Table 1 summarizes the three year response.

Table 1: Three Year Summary

	APIRE	APA	NASW	Total
Total Attendees	3,331	3,284	1,861	8,476
Total Returning Forms	1,520	2,545	1,740	5,805
Response Rate	46%	77%	93%	68%

CMHS is funding the MHCPE Program to continue to enhance the nation’s impact on the HIV/AIDS epidemic. For each of their 5 years of funding, each professional education site is expected to train 1,000 mental health professionals. They reach primary target audiences of psychologists, psychiatrists and social workers, all of whom play significant roles in treatment for individuals affected by HIV and AIDS. Each site utilizes their own site-specific curricula and the CMHS curricula to educate mental health providers on the neuropsychiatric, ethical, psychosocial and treatment aspects of HIV/AIDS. CMHS is seeking approval from OMB to continue conducting a systematic multi-site assessment of the education provided by the funded education sites. The multi-site effort logically builds on and extends their activities. This multi-site assessment will involve collecting information on the organization and delivery of the training sessions, as well as assessing the effectiveness of trainings. The multi-site feedback instruments collect descriptive information on each HIV/AIDS education training session using a Session Report Form to be completed by education site staff. Information on the effectiveness of the training as measured by participant satisfaction and increases in participant knowledge, skills, and abilities will be collected by feedback forms completed by participants. Participants attending sessions complete a single feedback form at the end of the training session. The education sites’ evaluators or their designees continue to be responsible for administering the instruments at training sessions. On a monthly basis, the education sites will submit the data, for processing and preliminary analysis, to the CMHS sub-contractor (for data processing and analysis). Table 2 summarizes the proposed multi-site data collection strategy.

Table 2: Summary of Over-all Data Collection Strategy

Curriculum	Feedback Form
	Participant Feedback Form	Participant Feedback Form (Neuropsychiatric Version)	Participant Feedback Form (Adherence)	Participant Feedback Form (Ethics)	SRF
General Education	X
Neuropsychiatric		X
Adherence			X
Ethics				X
SRF					X

2. Purpose and Use of Information

The information collected through the CMHS multi-site assessment effort benefits CMHS, the training sites, and the HIV/AIDS affected populations. The assessment data helps CMHS to continually improve and ensure high quality education programs that meet the needs of mental health providers serving those individuals most affected by the HIV/AIDS disease. This information also facilitates planning for future programs. For example, feedback from participants trained under prior years has helped CMHS to identify the need for additional education in specialized mental health issues.

The multi-site assessment activities are designed to help CMHS to fully describe the training sessions and participants served through the programs. CMHS uses the data collected under these programs to monitor the number of mental health providers attending training, participants’ demographic characteristics, and the effectiveness of training sessions. The data collected allows CMHS to understand the following organizational level issues:

• The characteristics of participants attending CMHS-funded sessions, which includes demographic characteristics, types of interactions with HIV-infected/affected individuals, primary work settings and extent of prior HIV-related experience;

• Topics covered at CMHS-funded trainings; and

• Educational methods employed to deliver the curriculum, which includes a description of the educational strategies used, material distributed, and involvement of HIV-positive individuals in training.

This information is important to CMHS for ensuring that the education sites are serving the intended populations of traditional and non-traditional mental health service providers, delivering training sessions that cover the breadth of topics specified in their contracts (general, neuropsychiatric, ethics, adherence and other curricula), and documenting the methods employed in delivering the various training sessions. Ultimately, this feedback helps both CMHS and the individual sites to continuously monitor and improve the education curricula, including their design, implementation and methodology.

The multi-site program assessment also provides a quality improvement mechanism to help individual sites to monitor the effectiveness of the tools used to deliver training, the organization of individual training sessions, and the training environment. The program assessment also allows CMHS to address individual level issues:

• The extent to which trainees are satisfied with the trainings they receive;

• The extent trainees indicate that attendance enhances their ability, willingness and comfort in working with HIV-infected/affected individuals;

• Trainee return rate for additional training or updates at education sites;

• The most effective types of trainings;

• Whether or not particular types of educational strategies and training delivery methods result in higher satisfaction levels than others;

• The characteristics of the education sites and sessions that are most effective in increasing trainees’ perceptions of enhanced work performance; and

• The characteristics of trainees who report greater satisfaction.

This project benefits CMHS, the education sites, and the HIV/AIDS service population in that it:

• Enables CMHS to monitor the quality of its education programs;

• Enables CMHS to assess the repertoire of skills and abilities of traditional and non-traditional mental health service providers;

• Allows CMHS to provide feedback and design technical assistance for funded education sites in order to improve efficiency and training effectiveness;

• Helps CMHS to ensure that the education programs are disseminating state-of-the-science information to HIV/AIDS mental health service providers, thereby enhancing services to service populations; and

• Guides CMHS in identifying model approaches to educating HIV/AIDS mental health service providers that can be widely disseminated.

Without this multi-site assessment, CMHS cannot empirically determine whether the funding of HIV/AIDS education is reaching the intended traditional and non-traditional mental health provider audiences. In addition, failure to conduct the program assessment would result in the diminished capacity of CMHS to provide targeted technical assistance to the education sites in order to improve the quality of education and training delivered. Without the assessment data, CMHS would lack the feedback needed to support continuous quality improvement and to ensure the needs of mental health providers and the HIV-affected populations they serve are being met, particularly minority populations. Failure to collect this information and ensure the efficacy of educational training for mental health providers would potentially result in diminished capabilities of service providers and lower quality of services for HIV/AIDS-affected populations.

3. Use of Information Technology

Procedurally, each of the education sites mails completed participant feedback forms to the CMHS evaluation subcontractor for data capture/entry/analysis. The evaluation subcontractor then electronically sends each education site a data file containing site-specific data for the respective site evaluators to use to complement their own local assessments.

The proposed multi-site data collection process increases the efficiency and practical utility of the assessment of these programs. The CMHS multi-site procedures and participant feedback forms were developed, tested and used in the MHCPE Program for over 10 years and demonstrated to work efficiently and effectively. The participant feedback forms and the procedures for electronic transmission of datafiles have been used and improved, based on program feedback, continually increasing the efficiency and minimizing the burden on both training participants and education site staff.

In addition, the statements of work for the professional associations require the development of World Wide Web-based educational training courses for their respective mental health professional audiences. The feasibility and cost associated with developing web-based versions of the participant feedback forms that trainees fill-out on-line when completing a web-based course continue to be explored. Offering trainees such an alternative might increase response rates, since trainees could complete the participant feedback forms electronically in real time immediately following the course. This option would also reduce keying costs since the data would be captured electronically. The cost has been deemed to be minimal and a pilot for the use of web-based feedback is under development.

4. Efforts to Identify Duplication

The data to be collected are unique to the CMHS HIV/AIDS education programs, are collected only for the CMHS programs, and are not available elsewhere. No other multi-site assessment activities are planned for the education sites. The data collected through the multi-site effort will be non-duplicative, minimize burden on respondents, and be of use to both CMHS and the education sites.

In its assessment design, CMHS has developed procedures to minimize burden on trainees who attend multiple MHCPE training sessions. Participants are asked to complete feedback forms to provide demographic information and feedback specific to each of the training sessions they attend. In the event that participants attend more than one MHCPE-supported training session, they are requested to complete the training-specific questions for each session, but are asked to complete the demographic information only once. The demographic information can then be mapped back to each training session for which the individual provides feedback information.

5. Involvement of Small Entities

This project will have no significant impact on small entities.

6. Consequences If Information Collected Less Frequently

The data is collected one time only from respondents attending CMHS-funded training sessions. Each trainee completes a participant feedback form only once near the end of a training session.

Failing to collect the information from all participants attending CMHS-funded educational training sessions would result in a missed opportunity by CMHS to fully describe the participants served under these education programs, and to conduct a comprehensive assessment of the effect of the education programs. The information provides a quality improvement mechanism for CMHS to continually monitor and refine its education programs to ensure they meet the needs of mental health providers. Without this information:

• CMHS would not be able to determine the extent to which it has helped to build a cadre of mental health providers, especially minority mental health providers;

• CMHS would not be able to monitor the quality of its education program and determine how it can be improved to ensure continued success at meeting the needs of mental health providers and the mental health needs of individuals with HIV and AIDS;

• CMHS would not be able to fully describe the range of mental health service providers being trained, and the representation of minority mental health service providers;

• CMHS would not be able to ascertain if participants are more knowledgeable about HIV/AIDS as a result of attending the education session; and

• CMHS would not be able to identify additional mental health service provider needs, including the potentially unique needs of minority mental health service providers.

7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with 5 CFR 1320.5(d) (2).

8. Consultation Outside the Agency

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on June 23, 2010 (75 FR 35819). No comments were received.

The multi-site design and participant feedback form design were based on initial consultation with experts in the field, and pilot testing. During the early stages of feedback form design, CMHS benefited from consultation with experts in the field of HIV training and education, design for collecting feedback, and feedback form development. Consultation with experts outside the agency was meant to minimize the burden on individual respondents and education site staff, to ensure the integrity of the form development, and to verify the appropriateness of the design for the program assessment. CMHS solicited input from consultants with expertise in HIV/AIDS, including clinical psychologists and psychiatrists, nurses, social workers, evaluation experts, HIV trainers, and directors of HIV/AIDS provider education programs. Input on the initial program assessment design and participant feedback forms was also solicited from four professional mental health provider associations that conducted HIV/AIDS education: the American Psychiatric Association, the American Psychological Association, the National Association of Social Workers, and the American Nurses Foundation. Additionally, as detailed in Section B4, a limited field test of the assessment design and instruments was conducted when the forms were initially designed for the MHCPE Program. The purpose of soliciting input from HIV/AIDS education site staff and participants was to gather feedback regarding the feasibility of the proposed multi-site program assessment and feedback forms. This initial feedback was used to modify the overall design and feedback forms to ensure consistency with ongoing training activities.

The assessment design and participant feedback forms have been used by MHCPE education sites for over 10 years. Current users of the forms have requested no revisions.

The assessment design and participant feedback forms were developed based on input from experts listed in Table 3.

Table 3: List of Experts Consulted

Experts Consulted Prior to the MHCPE II Program

John Anderson, Ph.D. American Psychological Association Office on AIDS (202) 336 – 6051	James Halloran, M.S.N., R.N., A.P.N. American Nurses Foundation (202) 651 – 7295
Charles Clark, M.D., MPH Florida Mental Health Institute (303) 442 – 6536	Carol Svoboda, M.S.W. American Psychiatric Association AIDS Program Office (703) 907-8668
Michael Dunham HI-Tech International, Inc. (703) 998 – 0287	Evelyn P. Tomaszewski, A.C.S.W National Association of Social Workers HIV/AIDS Spectrum Project (202) 408 – 8600, ext. 390
Michael Knox, Ph.D. Director, University of South Florida Center for HIV Education and Research Florida Mental Health Institute (813) 974 – 1925

Experts Consulted from the MHCPE II Program

John Anderson, Ph.D. American Psychological Association Office of AIDS (202) 336-6051	Cervando Martinez, Jr., M.D. University of Texas Health Science Center at San Antonio Department of Psychiatry (210) 567-4768
Francine Cournos, M.D. Columbia University (212) 543-5412	J. Stephen McDaniel, M.D. Emory University (404) 616-6310
Sally Dodds, Ph.D., LCSW University of Miami Department of Psychiatry & Behavioral Sciences (305) 355-9191	Ali Naqvi, Ph.D. Wayne State University AIDS Research and Education Program (313) 962-2000
Thomas Donohoe, M.B.A. UCLA Center for Health Promotion and Disease Prevention (310) 825-4750	Lisa Razzano, Ph.D. University of Chicago Mental Health Services Research Program (312) 422-8180, ext. 20
Abraham Feingold, Psy.D. (MHCPE II Steering Committee Chairperson) Boston, Massachusetts (617) 859-3953	Carol Svoboda, MSW American Psychiatric Association / Office on AIDS (202) 682-6104
Evelyn Tomaszewski, ACSW National Association of Social Workers (202) 336-8390

9. Payment to Respondents

Respondents will not receive any payments.

10. Assurance of Confidentiality

CMHS has designed the multi-site feedback data collection strategy so that no identifying information such as names or complete social security numbers will be requested of trainees. All feedback forms only request an eight-digit identifying number that is comprised of the last four digits of the respondent’s social security number and the month and day of their birth. This information is not specific enough to be considered a unique identifier, but will nevertheless enable CMHS to estimate the extent to which trainees attend multiple training sessions at specific sites. To further ensure confidentiality of individual responses, all data will be reported at the aggregate level so that individual responses cannot be identified; no data will be reported at the individual participant level.

11. Questions of a Sensitive Nature

No sensitive information will be requested in the multi-site participant feedback forms.

12. Estimates of Annualized Hour Burden

The total annualized burden for respondents for the Mental Health Care Provider in HIV/AIDS Education Program is estimated to be 1,843 hours.

The total burden to each of 10 potential respondent sites is estimated to be 184 hours. The total annualized hourly costs to Program participants across ten sites are estimated to be $4,713. The Center for Mental Health Services supports up to 10 HIV/AIDS education sites and each education site is required to provide training to at least 1,000 individuals per year. The estimates of annual hourly burden are therefore based on the assumption of 10 sites each serving 1000 participants per year. The burden estimates also assume that education sites will provide on average 5 training sessions per month or 60 per year.

All trainees attending the CMHS-funded training programs are asked to fill out an evaluation form at the end of the training session that is expected to take a maximum of 10 minutes to complete.

There is considerable diversity in the types of participants attending the training sessions and in their wage rates. Occupations range from physicians and nurses to outreach workers and clergy. For the purposes of calculating the total annualized cost, a wage rate of $25.00 per hour was used since the Program is intended to serve both traditional and non-traditional service providers. The burden estimates and resultant annualized costs are summarized below in Table 3.

The Mental Health Care Provider Education in HIV/AIDS Program is a continuation effort. This program consists of three associations and potentially seven grant supported education programs. All ten education sites are required to train a minimum of 1,000 mental health professionals per year using general, ethics, neuropsychiatric, neuropsychiatric for non-psychiatrists, and adherence curricula (all curricula are based on culturally competent mental health service provision). All sites have prior experience in providing HIV/AIDS related mental health training to traditional and non-traditional mental health providers. Each education site conducts about 60 trainings per year. Each site conducts the following types of training sessions: about 25 using the general curriculum, 12 using the neuropsychiatric curriculum for non-psychiatrists, 10 using the ethics curriculum, 8 using the neuropsychiatric curriculum, and 5 using the adherence curricula. The appropriate participant feedback form will be administered to trainees after each session.

Table 4: Annual Burden Estimate

Annualized Burden Estimates and Costs
Mental Health Care Provider Education in HIV/AIDS Program (10 sites)

Form

Number of Respondents

Responses Per Respondent

Total Responses

Hours per Response

Total Hour Burden

Hourly Wage Cost

Total Hour Cost ($)

All Sessions

One form per session completed by program staff/trainer

Session Report Form

600

1

600

0.08

48

$25.00

$1,200

Participant Feedback Form (General Education)	5000	1	5000	0.167	835	$25.00	$20,875
Neuropsychiatric Participant Feedback Form	4000	1	4000	0.167	668	$25.00	$16,700
Adherence Participant Feedback Form	1000	1	1000	0.167	167	$25.00	$4,175
Ethics Participant Feedback Form	2000	1	2000	0.167	125	$25.00	$3,125
Total	12,600		12,600		1,843		$46,075

13. Estimates of Annualized Cost Burden to Respondents

No capital or start-up costs are involved nor is there any cost to respondents or record keepers resulting from the collection of information.

14. Estimates of Annualized Cost to the Government

The average annual estimated cost to the Federal Government for the multi-site program assessment is $395,000 for the 5-year MHCPE Program. This includes the costs associated with collecting feedback data, multi-site assessment and information dissemination. CMHS will fund ten education sites. For the purposes of calculating the annualized cost to the government, it is estimated that each education site will devote approximately 10% of their average annual award to multi-site assessment activities. Per site of the 10 sites, annual multi-site assessment-related costs are expected to be $18,500 for a total of $185,000, for conducting assessments with 1,000 participants each year/site. It is estimated that approximately $200,000 will be spent annually for overseeing the multi-site program assessment, processing and analyzing data, and preparing reports for their respective education sites. An additional $10,000 per year in Government monitoring costs, including travel, is anticipated. The total per year cost estimated for this program is estimated to be $395,000.

15. Changes in Burden

There is no change in burden.

16. Time Schedule, Publication and Analysis Plans

The education sites in the MHCPE Program are funded for a period up to 5 years with annual awards being made subject to the continued availability of funds and progress achieved. The current program began its first funding cycle on approximately September 30, 2009. A request for approval of use of the participant feedback forms is being re-submitted to OMB, now, at the ending of the first program year.

Data collection will continue after CMHS has received OMB clearance for use of the proposed assessment design and participant feedback forms. Education sites will receive a PDF version of the newly approved OMB forms for their use.

Education sites will mail completed forms to the CMHS subcontractor for data capture/entry. The contractor will then send electronic datafiles and return the original feedback forms to the education sites.

The mental health professional association contractors are required to submit quarterly progress reports to CMHS. Additional specialized reports may be required.

Table 4 shows the major activities of the professional association education sites, and the anticipated dates of completion.

Table 5: Projected Schedule of Activities and Timelines

Major Activity		Dates
YEAR 1
1. MHCPE Programs (Contracts) funded for continuation		September 2009
2. Multi-site program assessment procedures (continue)		November 2009
3. Multi-site feedback data collection continues		November 2009(with approval January 2011)
4. Education sites submit feedback forms to CMHS subcontractor		Monthly (with approval January 2011)
	5. CMHS subcontractor continues to sends quarterly reports to their respective education sites		Monthly (with approval February 2011)
	6. Education sites continue to send quarterly reports to CMHS		Quarterly (effective March 2008)
	7. All sites submit annual report to CMHS		October 2011
	YEARS 2 and 3
	1. Education sites submit feedback forms to CMHS subcontractor		Monthly
	2. CMHS subcontractor sends quarterly reports to their respective education sites		Quarterly
	3. Education sites send quarterly reports to CMHS		Quarterly
	4. All sites submit annual report to CMHS		October (yearly)
	5. Final report from CMHS subcontractor		December 2014

On a monthly basis, the education sites submit the multi-site participant feedback forms to the CMHS subcontractor for processing. Upon receipt of the feedback forms, the forms are briefly reviewed to ensure that information to be manually entered (e.g., session number and date, training and education site number) has been recorded. Forms then are keyed, and electronic datafiles are produced and electronically mailed to each education site. Each site receives its own data. The education sites will receive electronic copies of their data on a monthly basis. These reports may contain descriptive statistics such as measures of central tendency including means, medians, modes, variances and standard deviations. Table 6 contains a data analysis plan that shows the major study questions, instrument items and types of analysis used to answer the questions at the end of the program.

In addition, the CMHS subcontractor produces quarterly and annual reports on the aggregated data, across sites, for CMHS use in program monitoring.

The CMHS Government Project Officer may also request special focused analyses. Among the statistical techniques that may be employed in producing special reports or publications are descriptive statistics, regression or logistic regression depending on the dependent variable, analysis of variance, t-tests and outlier analyses. These reports and publications also may also be presented at periodic meetings as well as regional and national conferences.

17. Display of Expiration Date

The expiration date will be displayed.

18. Exceptions to Certification Statement

There are no exceptions to the certification statement.

Table 6: Data Analysis Plan

Organization and Delivery of the Training

Program Assessment Question	Items on Instrument	Types of Analyses
1. Characteristics of participants attending trainings.	Number of participants in session; demographic data; primary work settings; number of years provided services.	Descriptive statistics: Frequencies and Measures of Central Tendency
2. Topics covered by individual sites and across the Program.	Topics covered during training (e.g., epidemiology of HIV/AIDS, substance abuse issues, adherence to treatment).	Descriptive statistics: Frequencies and Measures of Central Tendency
3. Training methods used at education sites.	Type of curriculum used (general, ethics, neuropsychiatric); workshop length; training delivery method.	Descriptive statistics: Frequencies and Measures of Central Tendency

Impact of Training

4. Were the trainees satisfied with the trainings?	Questions on the organization of the training session and the usefulness of information/skills training.	Inferential statistics: Paired t-tests, ANOVA
5. Did trainees indicate that attendance enhanced their ability, willingness and comfort in working with HIV-infected/affected individuals?	Willingness to treat and/or care for HIV-positive/affected individuals; comfort working with HIV-positive/affected individuals; capability in treating and/or caring for HIV-positive/affected individuals.	Inferential statistics: Paired t-tests, ANOVA
6. Did trainees return to sites for additional training or updates?	Received any additional HIV/AIDS-related education since attending training session.	Descriptive statistics: Frequencies and Measures of Central Tendency
7. Were some types of trainings more effective than others?	Types of curriculum used; satisfaction with training; knowledge gained from training.	Chi Square Test of Significance; Content analysis of open-ended comments
8. Do particular types of educational strategies and training delivery methods result in higher satisfaction levels than others?	Types of Strategies/methods employed; type of curriculum used.	Regression Analysis; Content analysis of open-ended comments from trainees
1. What are the characteristics of education sites and sessions that are most effective in increasing trainees’ perceptions of enhanced work performance?	Type of curriculum used; involvement of HIV+ individuals in training; strategies/methods employed; materials distributed.	Regression Analysis; Content analysis of open-ended comments from trainees
2. What are the characteristics of trainees who report greater satisfaction?	Demographic data; type of curriculum used.	Regression Analysis

11. B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

CMHS funds three mental health professional associations that are required to train approximately 1,000 professionals each year (potentially a total of 10 education sites may be funded). Therefore, approximately 3,000 (10,000) participants will be trained and a census of these will be asked to complete feedback forms annually. A review of the types of mental health providers served by previously funded education sites suggests that a wide range of participants can be expected. Types of trainees are likely to include social workers, psychologists, psychiatrists, nurses, clergy, counselors, non-psychiatric physicians, and other health and non-health care workers.

All training participants will be asked to provide feedback on the training sessions they attend. No sampling procedures will be employed. This approach is consistent with other types of program assessment for education programs and is consistent with the approach used in the earlier with the CMHS HIV/AIDS education program. The reasons for collecting feedback on the entire population include:

• The importance of fully assessing which types of mental health providers attend the training sessions, their demographic characteristics, the types of HIV-related services they provide and the types of HIV- and AIDS-affected clients they serve;

• The limited number of education sites to be funded;

• The diversity of sites in terms of geographic location and primary affiliation (e.g., university vs. community based);

• The differing training venues at which trainings are delivered;

• The differing training methods that are employed; and

• The variation in topics likely to be covered in the training sessions.

In order to maximize response rates, the participant feedback forms were designed to collect the minimum amount of information necessary for CMHS to address the assessment questions. Additional specific considerations for maximizing responses are discussed in Section B3. A limited review of currently operating education sites indicates that the response rates vary by training session size and venue, with smaller training sessions having higher response rates. Based on the experience of the earlier CMHS MHCPE Programs, the overall estimated response rate remains high with most sites, near or over 80%, with one that is lower, due to the specific context of the in-hospital grand-rounds training setting. The information collected will be used to solicit feedback for improving the training sessions and HIV/AIDS education program, as well as to get feedback on the usefulness of the education training to participants. During the training meeting, education site staff and other designees receive instructions from the CMHS Government Project Officer and the CMHS subcontractor on the administration of the participant feedback forms and the submission of forms for processing.

2. Information Collection Procedures

Feedback will be collected from all participants that attend training sessions conducted under the MHCPE Program. Participants will be asked to complete feedback forms based on the type of training session they attended. Table 1 in Section A summarizes the overall data collection strategy. The data collection strategy proposed for use will be the same as the strategy used in the current MHCPE Programs. The No revisions are requested for the program assessment design..

As illustrated in Table 1, education site staff will complete a Session Report Form that describes the training environment for all training sessions delivered. All trainees will be asked to complete a participant feedback form at the end of the training session.

For over 10 years of the MHCPE Program (beginning Sept. 1999), and the current 5 year cycle beginning September 2009, the CMHS Office’s subcontractor collects and processes the feedback forms. The subcontractor returns (via e-mail) electronic versions of the scanned data and distributes data-based reports to education sites on a quarterly basis.

3. Methods to Maximize Response Rates

A limited field test of procedures and instruments was conducted at the inception of the CMHS MHCPE II Program to assess the feasibility of administering that multi-site effort. One of the specific aims of the field test was to solicit information from education site staff, trainers, evaluators and training participants regarding methods for maximizing the response rates. One component of the field testing procedures involved administering the instruments to less than 10 training participants and then conducting a focus group to solicit comments regarding: (1) the likelihood of obtaining responses for selected items, and (2) methods of increasing the response rate of survey respondents. This process yielded valuable information that CMHS has implemented to maximize response rates and the usefulness of the information requested. To maximize response rates, CMHS, in the previously funded MHCPE Programs:

1. Ensured that the questions on the multi-site feedback forms are the minimum needed to address the CMHS research questions;

2. Ensured that the multi-site feedback forms were as user-friendly as possible and contained easy-to-read font, logical layout and straightforward language;

3. Provided clear instructions for all sections of the surveys;

4. Used culturally sensitive questions that are unlikely to be perceived as offensive or compromising to the respondents’ values and belief systems; and

5. Used culturally sensitive questions that are unlikely to be perceived as offensive or compromising to the respondents’ values and belief systems.

The MHCPE Program has used the participant feedback forms for over 10 years.

4. Tests of Procedures

Prior to their use in the MHCPE Program, the initial assessment design and participant feedback forms were pilot-tested on a small sample of less than 10 individuals to ensure that the multi-site assessment requirements and procedures were consistent with activities conducted at education sites. The field-testing was designed to collect information on the overall evaluation design and draft feedback forms. Comments on the draft feedback forms included collecting information on the likelihood of obtaining specific responses, overall instrument layout, item flow, and administration times. Feedback on the overall design included collecting information on the:

1. Overall feasibility of administering feedback forms at sessions of varying lengths;

2. Feasibility of trainers and staff administering feedback forms;

3. Anticipated challenges in submitting information to Coordinating Center;

4. Identification of activities required to coordinate multi-site data collection activities at the local level;

5. Recommendations for alleviating the data collection burden;

1. Recommendations for ensuring that the feedback forms are gender, age and culturally sensitive; and

2. Recommendations for improving overall design for soliciting feedback from participants.

As discussed in Section B3 and based on the feedback provided to CMHS by the MHCPE education sites no revisions are requested.

CMHS is proposing the continued use of the post-session only participant feedback forms as currently in use, in the current post-session feedback design, as approved by OMB in 2004. The current assessment design and the participant feedback forms are providing CMHS with invaluable information to inform quality improvement efforts. Further, the assessment data enables CMHS to monitor progress in meeting programmatic goals of educating providers of mental health services for HIV and AIDS-affected individuals and enhancing the nation’s ability to have an impact on the HIV/AIDS epidemic.

5. Statistical Consultants

The names and phone numbers of statistical contacts, individuals responsible for collecting and analyzing the data and responsible agency personnel are provided below, in Table 7.

Table 7: Data Collection Personnel, Analysts, Statistical Consultants

and Responsible Agency Personnel

Education Site Directors
Organization	Contact	Title	Telephone
American Psychological Association	John Anderson, Ph.D.	Project Director	(202) 336-6051
American Psychiatric Association	Diane Pennessi, M.D.	Project Director	(703) 907-8668
National Association of Social Workers	Evelyn Tomaszewski, ACSW	Project Director	(202) 336-8390

Statistical Consultants
Organization	Contact	Title	Telephone
James Bell & Associates, Inc.	James Bell	Project Officer	(703) 528-3230
Agency Personnel Responsible for Deliverables

Organization	Contact	Title	Telephone
CMHS	Ilze L. Ruditis, MSW	Government Project Officer, CMHS	(240) 276-1777

LIST OF ATTACHMENTS

• Attachment A: Instructions for Administering Forms, Session Report Form and Multi-Site Feedback Forms

NOTE TO OMB REVIEWER

The participant feedback forms submitted for OMB approval have been used for the over 10 years in the CMHS MHCPE Programs.

The participant feedback forms contain several sections that are identical across the different types of forms in order to enable data to be pooled and compared across sites and types of training sessions. Detailed explanation of the similarities and differences across the forms is provided below to facilitate the review.

1. The instructions for completing all participant feedback forms are the same except for the estimated completion time, which varies according to the instrument’s length.

2. For all of the Participant Feedback Forms (i.e., General, Adherence, Neuropsychiatric, Ethics) items on the first page (items 1 through 11) are the same.

3. For the General, Adherence and Neuropsychiatric Participant Feedback Forms, items 12 through 19 are identical.

4. The Session Report Form contains completely unique items. None of its items appear on the participant feedback forms.

Attachment A:

Instructions for Administering Forms, and

Multi-Site Feedback Forms

A-1 Instruction for administering the Session Report Form

A-2 Session Report Form

A-3 Instructions for administering the Participant Feedback Form – General Education

A-4 Participant Feedback Form – General Education

A-5 Instructions for administering the Participant Feedback Form (Spanish Version) – General Education

A-6 Participant Feedback Form (Spanish Version) – General Education

A-7 Instructions for administering the Participant Feedback Form (Neuropsychiatric Version)

A-8 Participant Feedback Form (Neuropsychiatric Version)

A-9 Instructions for administering the Ethics Participant Feedback Form

A-10 Ethics Participant Feedback Form

A-11 Instructions for administering the Participant Feedback Form (Adherence Curriculum)

A-12 Participant Feedback Form (Adherence Curriculum)