MFP_OMB__supporting_statement_092409

Evaluation of SAMHSA’s Minority Fellowship Program (MFP)

OMB Submission

THE SUPPORTING STATEMENT

A. Justification.

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS), has funded an Evaluation of the SAMHSA Minority Fellowship Program (MFP). This evaluation will use the following six instruments:

• Survey of Current SAMHSA MFP Fellows

• Survey of MFP Alumni

• Survey of Current and former members of Selection and Advisory Committees

• Telephone Interview Protocol for SAMHSA MFP Staff and other SAMHSA officials

• Telephone Interview Protocol for Current and Former Program Directors in the Fellowship Program; and

• Telephone Interview Protocol for organizations administering the Minority Fellowship grant from SAMHSA.

In 1974, in response to a substantial lack of ethnic and racial minorities in the mental health professions, the Center for Minority Health at the National Institute of Mental Health established the Minority Fellowship Program (MFP). The MFP initially was funded through NIH but under the ADAMHA Reorganization Act (P.L. 102-321), the MFP moved to SAMHSA in 1992. The MFP is supported by funds from all three SAMHSA centers: the Center for Mental Health Services (CMHS), the Center for Substance Abuse Treatment (CSAT), and the Center for Substance Abuse Prevention (CSAP).

The MFP facilitates the entry of minority graduate students and psychiatric residents into mental health careers and has increased the number of psychology, psychiatry, nursing, and social work professionals trained to provide mental health and substance abuse services to minority groups. These four disciplines train many of the nation’s professionals in mental health and substance abuse treatment and prevention. Up until FY 2007, grantees have been limited to the American Nurses Association (ANA), the American Psychiatric Association (ApA), the American Psychological Association (APA), and the Council on Social Work Education (CSWE). Through the MFP, SAMHSA has awarded funds to the four grantees (ANA, ApA, APA, and CSWE) to provide fellowships and programmatic support to over 570 doctoral level graduate students and psychiatric residents since 1992.

Minority Fellowship Program grants are authorized under Section 520A of the Public Health Service Act, as amended and the program currently addresses two Healthy People 2010 focus area(s) 18 (Mental Health and Mental Disorders) and 26 (Substance Abuse).

Consistent with federal efforts to address issues of accountability, capacity and effectiveness (such as the Program Assessment Rating Tool (PART)), SAMHSA has undertaken a three stage approach to independently evaluate the MFP: 1) develop a logic model, 2) prepare an Evaluation Plan, and 3) conduct the evaluation and write a report of findings and recommendations. SAMHSA evaluation studies are authorized by Section 501 (d) (4) of the Public Health Service Act (42 USC 290aa).

This evaluation requests approval of six data collection instruments (to be used with approximately 1015 participants), all of which are included in the section of this application entitled “Survey and Interview Protocols. Three on-line (Internet-based) surveys (with the option for a hardcopy mailed through U.S. postal service) will be used with the following stakeholders in the MFP:

1. Current SAMHSA MFP Fellows selected by the four grantee programs (Attachment A). Fellows currently receiving support during their doctoral level training or psychiatric residency will be asked about their experiences in the MFP (from recruitment into the program through their participation in the various activities provided by the ANA, ApA, APA and CSWE).

2. MFP Alumni who participated in the four programs during the time the program was administered by SAMHSA (Attachment B) will be asked about their previous experiences as Fellows in the MFP as well as their subsequent involvement and leadership in their professions.

3. Current and former members of Selection and Advisory Committees in each of the four grantee programs (Attachment C) will be surveyed about the procedures each of the grantees uses to select and monitor Fellows.

Finally, the following three instruments (each to be used with 9 or fewer participants) have been included in the OMB package to provide context for the entire evaluation:

1. Current and former SAMHSA MFP Staff and other SAMHSA officials involved in the MFP will be asked about a) the federal goals for the MFP, b) the federal supports provided to the grantees, and c) their assessment of the MFP. Note that these participants also will be asked to fill out a form prior to the interview. Both documents are included in this application for review. (Attachment D)

4. Current and former MFP Program Directors or Senior Staff in each of the four grantee programs will be interviewed about how they operate the MFP including a) how Fellows are recruited, selected and supported during the time the Fellows are in the program; b) how the grantees monitor and report their own activities in administering the SAMHSA grant; and c) their assessment of the MFP. Note that these participants also will be asked to fill out a form prior to the interview. Both documents are included in this application for review. (Attachment E)

5. Staff in each of the grantee’s host organizations (i.e. staff in the ANA, APA, ApA, and CSWE) will be interviewed (Attachment F) about the organizational context which the host organization provides to the MFP grant (e.g. tangible Contributions and Resources such as office space as well as networking opportunities for career development).

All SAMHSA grantees will collect and report certain data so that SAMHSA can meet its obligations under the Government Performance and Results Act (GPRA).  Each year, grantees submit data on four GPRA performance indicators through yearly progress reports on:

1. Employment of Past Fellows: A tabular listing of the names of Fellows who have completed the program and the types of jobs they held, especially those related to providing mental health/substance abuse services to underserved populations, indicating whether the job setting is public or private.

2. Attrition of Students from Training Programs: Using the total number of Fellows in each program for the current year as a basis for comparison, the number/percent of students (by year, gender, ethnicity) are reported in the following categories: those who graduated during this period, those terminated before graduation and reasons why, and those who may have become disabled.

3. Descriptions of Current Fellows: The names of Fellows by university, ethnicity, and gender, including both those who were newly admitted to the program during this period, as well as those previously admitted.

4. Assessment of the Program: Any program-specific indicators of success, especially program assessment data grantees have collected or have pending as a way of measuring the goals and/or achievements of their program.

While this information request is consistent with those efforts in that the evaluation will assess program performance, the data collection efforts proposed do not duplicate these efforts and the proposed data to be collected are not available elsewhere (including from the MFP Coordinating Center).

2. Purpose and Use of Information

The practical utility of this evaluation is to comply with progress reporting. This independent evaluation is the first one in the program’s history, and these surveys constitute the predominance of information to be gathered about the MFP for the evaluation.

A logic model which was constructed, based on input from SAMHSA program staff and the evaluation’s two advisory panels (Subject Matter Experts and Advisory Panel), has been included (Figure 1). This model has been used to guide the construction of research questions.

Data to be collected in this evaluation are limited to the activities and functions most directly linked to SAMHSA’s Minority Fellowship Program, depicted in the shaded areas of the logic model. Therefore, while recognizing their importance, the evaluation will not be assessing the range of historical and contextual influences on the program (left hand portion of the model); the processes used by universities and medical institutions to locally implement the Fellowship programs (lower middle section of the model); or the more distal outcomes (right hand portion of the model). These influences and outcomes are both less easily assessed and less directly amenable to modification by SAMHSA.

The central portion of the logic model, however, will be used to guide the evaluation. Examples of the kinds of information to be collected, analyzed and distributed include information about the processes involved in implementing the MFP at both the federal and grantee organization levels. These processes begin with the a) federal formulation of SAMHSA’s priorities which provide the context for issuing the program guidance to applicants and proceed through b) the federal review of applications, c) federal funding of four MFP grantees, d) the implementation of the grants by the four professional associations, e) federal overview of that implementation, and f) grantee processes which document the performance and accountability of the SAMHSA MFP in their discipline. In addition, the evaluation will assess outcomes of the MFP which correspond to those articulated in the original and current grant announcements for the MFP, namely the ability of the MFP to

• Develop and support a new cohort of Fellows;

• Train/ mentor ethnic/racial minority students and professionals in mental health/substance abuse treatment;

• Increase the number of ethnic/racial minority professionals in mental health/substance abuse treatment;

• Increase diversity in mental health/substance abuse leadership;

• Increase professional contributions in mental health/substance abuse treatment for minority populations;

• Increase institutional involvement of ethnic/racial minority professionals in the areas of mental health and substance abuse treatment; and

• Increase mental health and substance abuse services to minority communities.

SAMHSA’s MFP is but one influence on the distal outcomes which assess more complex outcomes (such as transformation of the workforce). Since it would not be possible to directly attribute any changes in these outcomes to the MFP, this evaluation will not assess such outcomes.

Furthermore, the evaluation will be descriptive in nature and given the nature of data to be collected, no statements will be made about whether the Minority Fellowship Program caused the proximal outcomes.

Where practical, results will be interpreted in light of policy changes which SAMHSA and/or the grantees could implement to improve program functioning.

Findings from the evaluation will be disseminated in a way that addresses the various needs of multiple stakeholders and maximizes the multiple uses to which the findings may apply. The range of stakeholders who have active interests in the MFP include:

• MFP program staff in SAMHSA who are charged with implementing the program, monitoring adherence, and quality improvement;

• Other federal agencies with substantive or evaluative interests overlapping with those of the MFP;

• The four professional associations originally included under the funding mechanism: the American Nurses Association (ANA), American Psychiatric Association (ApA), American Psychological Association (APA), Council on Social Work Education (CSWE) to promote program improvement at the organizational level; and

• The fifth professional association added in FY 2007 to the MFP (American Association for Marriage and Family Therapy (AAMFT)) also to promote program improvement at the organizational level. Although this association is not included in the evaluation, the information gathered here will be relevant for their program.

Additional stakeholders also have active interests in the MFP, including:

• Executive and legislative staff who are concerned with oversight and management, including how the MFP fits in with other DHHS programs and the degree to which the program contributes to the achievement of broad policy goals;

• Mental health and substance abuse treatment professionals;

• Academic training programs in mental health and substance abuse treatment;

• Psychiatric residency training programs; and

• Consumers and allied advocacy groups who are concerned about culturally competent mental health and substance abuse services.

3. Use of Information Technology

To minimize the burden to the MFP grantees and other stakeholders, existing data which already is available will be gathered from CMHS and each of the four grantees. New data collection will be gathered using a mixed media approach that is planned to improve data quality, agency efficiency, and responsiveness to the public using two approaches:

• On line surveys of three groups:

  • Current SAMHSA MFP Fellows

  • MFP Alumni

  • Current and former members of Selection and Advisory Committees

• Telephone interviews of three additional groups:

  • SAMHSA MFP Staff and other SAMHSA officials involved in the development of the MFP

  • Current and Former Program Directors operating the grants in the ANA, ApA, APA, and CSWE

  • The Host organizations (ANA, ApA, APA, and CSWE) overseeing the MFP grant from SAMHSA.

The contractor will obtain contact information from the Coordinating Center overseeing the grantee programs (ANA, ApA, APA, and CSWE) in order to send an initial letter (via the U.S. Postal Service) and/or e-mail to those who will be asked to participate in the on-line survey (Attachment K; Attachment L). In this initial correspondence, participants will be provided with a log-in and password which can be used to complete the on-line survey. As an alternative, participants also will be provided with contact information to request a hardcopy questionnaire if they do not wish to use the electronic survey.

All web based surveys will comply with the Section 508 of the Rehabilitation Act to permit accessibility to people with disabilities. Surveys to be delivered using the electronic web surveys will be designed to ensure that they can be completed through multiple sessions/ log-ins which permits the participants to accommodate the data collection request in a way that is respectful of their schedules.

The contractor already has contact information for individuals who will be asked to participate in telephone interviews. Since SAMHSA staff and the Program Directors of the grants are aware of the evaluation, the contractor will directly telephone these individuals to request an interview (Attachment J). An introductory e-mail will be sent to the individual in the host organization who oversees the Program Director which explains the evaluation and the purpose of the interview; approximately one week later, a telephone call will be made to schedule the interviews at a later time.

4. Efforts to Identify Duplication

The data collection proposed for this evaluation is not available elsewhere, is not duplicative, and is seen as critically valuable for assessing the effectiveness of SAMHSA’s Minority Fellowship Program.

5. Involvement of Small Entities

The information collected does not significantly involve small entities.

6. Consequences if Information Collected Less Frequently

Each selected respondent is asked to respond one time only. Participation is voluntary. If participants were not included or information were not collected from those selected for this study, then the information collected could present a biased picture of the MFP.

7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required in 5 CFR 1320.8(d) was published in the Federal Register on December 5, 2008 (Volume 73, Number 235, Pages 74177-74178].

No comments were received.

Feedback from External Project Advisors

The Evaluation Plan containing all data collection instruments has been reviewed by SAMHSA staff, and two external advisory groups constituted specifically for this evaluation: an Advisory Panel (AP), made up of 7 members representing program stakeholders of diverse backgrounds, interests, and 4 Subject Matter Experts (SME) who either received a Fellowship from the MFP or have participated in some other way within one of the four grantee programs (e.g. as an Advisory Panel member) but are independent from its administrative operations.

Consultation with the SME and the AP is ongoing and explicitly sought by the project at significant stages of this evaluation. SAMHSA and both panels assisted in designing the logic model of the MFP and reviewing the research questions which guided the design of the survey instruments and telephone interview protocols. Additionally, these panels reviewed the survey instruments and telephone interview protocols.

9. Payment to Respondents

No payments or gifts are provided to respondents for their participation in this data collection.

10. Assurance of Confidentiality

Participation in this evaluation is voluntary. All survey introductions include the purpose of the information collection and that this activity is sponsored by the Federal Government. Additionally, survey introductions inform respondents that their information will be kept private, and further that they are free to skip any question that they do not wish to answer. None of the data to be collected are protected health information (“PHI”).

Finally, SAMHSA will follow policies and procedures to ensure the security and privacy of all data. No direct identifiers (e.g. name, address, telephone numbers, etc) will be included in the data set. Furthermore, privacy will be maintained by using a Web-based data entry and file transfer system that uses industry standard secure socket layer data (SSL) encryption; firewall protection against unauthorized access to data; Web access that requires use of assigned user names and passwords; data files that include person-level information that are password protected with access limited to only those individuals who have a need to work on them using a secure decryption key; and the data will be stored on a secure partition of a dedicated Windows-based server and as such will be strongly encrypted. Access to data on this server (from both inside and outside the data center) is username and password protected. 

11. Questions of a Sensitive Nature

There are no questions of a sensitive nature. All surveys provide assurances that respondents have the right to refuse to answer any questions, and that the information they provide is private.

12. Estimates of Annualized Hour Burden

The maximum yearly burden for the project is based on analysis of the amount and type of data requested, using previous experience with similar data gathering activities, and is presented in Table 1.

Table 1: Estimate of Burden
Form Name	Number of Respondents	Responses per Respondent	Hours per Response	Total Hours
Current SAMHSA MFP Fellows Survey	100	1	1.5	150
SAMHSA MFP Alumni Survey	850	1	2	1700
MFP Selection and Advisory Committees Survey	40	1	1.5	60
Current and former SAMHSA MFP Program Staff and other SAMHSA officials Interview Protocol (and form)	8	1	2	16
Current and former MFP Program Directors or Senior Staff (from the grantee organizations) Interview Protocol (and form)	8	1	2	16
Grantee host organization Interview Protocol	8	1	1	8
Totals	1014	-----	-----	1950

The estimated response time ranges between 1 and 2 hours per participant to either complete the questionnaires via the Internet (including produce and submit an electronic file) or to answer questions over the telephone. Variation in burden may occur due to the complexity of interventions being implemented, organization of records, the availability of informed staff to complete the instruments, and other unpredictable factors.

13. Estimates of Annualized Cost Burden to Respondents

There are no capital, start-up, operations, maintenance or purchase of services costs that exceed standard business expenses associated with participating in this evaluation.

Costs associated with the time of current fellows (graduate students and psychiatric residents) are estimated using $20 per hour. Given the variability in wages of former fellows as well as of Advisory/Steering Committee Members, $30 per hour has been used for these two groups. Other wage estimates have been calculated using national averages from the U.S. Census Occupational Employment and Wages, May 2006 (www.bls.gov/oes/current/oes_nat.htm) using the following categories:

Stakeholder Group	Census Data Categories
Current and Former MFP Program Directors	Educational Administrators, post-secondary (SOC 11-9033) in Colleges, Universities and Professional Schools (NAICS 611300)
Grantee Host Organizations	Managers (SOC 11-000) in Business, Professional, Labor, Political, and Similar Organizations (NAICS 813900)

Estimates of annualized cost to the respondents also are included in Table 2.

TABLE 2: Estimate of Cost

Surveys	Number of Respondents	Hours per response	Average Hourly Wage	Total Hour Estimate for Wages
Current SAMHSA MFP Fellows Survey	100	1.5	No direct costs to participants – volunteers. Used $20/hour for current fellows; $30/hours all others	$300
SAMHSA MFP Alumni Survey	850	2		$5100
MFP Selection and Advisory Committees Survey	40	1.5		$1800
Subtotal: Protocols with more than 9 participants	990	---	---	$7200
Current and former MFP Program Directors or Senior Staff (from the grantee organizations) Interview Protocol	8	2	$101,950/ 2080=$49.01	$784.16
Grantee host organization Interview Protocol	8	1	$92,710/ 2080= $44.57	$356.56
Subtotal: Protocols with less than 9 participants	16	---	---	$1140.72
Totals	1006	-----		$8,340.72

14. Estimates of Annualized Cost to the Government

SAMHSA/CMHS has planned and allocated resources for the efficient and effective management and use of the information to be collected, including the processing of the information in a manner which shall enhance, where appropriate, the utility of the information to agencies and the public. The total cost of this evaluation, being performed under a Task Order, is $799,346. Initial activities, including a report using existing data sources already have been completed. The remaining budget of $ 384,636 has been allocated to the design, data collection and reporting activities outlined in this application.

SAMHSA /CMHS estimates the GS-13 Government Project/Task Order Officer (GPO/TOO) principally involved in the oversight and analysis of this contracted evaluation has spent on average approximately 2% of his time (1 hour weekly) overseeing various components of this project. On an annualized basis this would be the equivalent of $1460 in federal employee personnel costs (based on an annualized GS-13 salary of $72969) over the last two years that the project has been underway.

The contractor has been reimbursed on a monthly basis for operational expenses consistent with the terms of their cost-reimbursement contract.

15. Changes in Burden

This is a new project.

16. Time Schedule, Publication and Analysis Plans

The current contract, including a no-cost extension, ends September 29, 2009. A final report is to be delivered by this date to CMHS. Evaluation activities have been stated in Table 3 below relative to the time OMB approval is secured:

Table 3: Timeline for New Data Collection Activities

	Months After OMB Approval
	01	02	03	04	05	06	07	08	09
Revise Instruments	X
Telephone Interviews
Schedule Interviews	X	X
Conduct Interviews		X	X	X
Code Data			X	X	X
Enter Data into Analysis File				X	X
On-Line Surveys
Coordinate and Distribute Letter of Invitation from Grantees	X	X
Distribute E-mail instructions	X	X
Follow-up #1 Reminder with e-mail link		X	X
Follow-up #2 with hardcopy		X	X	X
Extract Data from On-Line			X	X	X
All Data
Clean			X	X	X	X
Merge/Create Analysis Files				X	X	X	X
Analyze Data					X	X	X	X
Write/Revise Reports					X	X	X	X	X

Analysis Plan

Using a logic model developed in consultation with the MFP Program Directors and the Evaluation’s Advisory Panel (See page 4, Figure 1), specific evaluation questions have been constructed and grouped into five modules:

1. Federal resources devoted to the program (Federal inputs)

2. Grantee resources devoted to the MFP (grantee inputs)

3. Implementation of the MFP at the Federal level (federal processes and outputs)

4. Implementation of the MFP at Grantee organizations (grantee processes and outputs)

5. Outcomes (proximal outcomes)

The specific questions to be answered by this evaluation include:

Module A: Federal Resources

1. Federal Priorities

1. What Program Priorities has SAMHSA identified agency-wide, during the years it has operated the MFP, which are relevant for the MFP?

2. Proposal Review

1. To what degree does the federal review process support grantees efforts to improve over their prior performance?

2. How does the review process of the MFP support SAMHSA’s priorities?

3. Tangible resources provided by SAMHSA

1. What level of support has SAMHSA provided the MFP, over the years? Including staffing, external technical assistance, and grantee funding?

2. Are these levels sufficient to support the MFP?

Module B: Grantee resources devoted to the MFP

1. Organizational Context

1. What is the relationship of the four grantee programs to their respective host organizations?

1. Is the MFP grantee program physically located in the host organization’s building?

2. What resources (including (but not limited to) staff, meeting space, overhead) does the host organization provide to the MFP program?

3. How do the networks of the host organizations foster the goals of the Minority Fellowship Program?

Module C: Implementation of the MFP at the Federal level

1. Grants Management

1. How does the SAMHSA grants management process influence the grantee programs?

2. How competitive are the grant management and review processes?

2. Program Monitoring

1. How many site monitoring visits have occurred?

2. How is information from these visits used to manage the MFP?

3. How effective are the site visits?

4. The grantees are required to submit various types of information to the federal program office: plans, reports and some data elements on the fellows (GPRA). How is this information used in managing the federal program?

3. Technical Assistance

1. Is TA provided and, if so, and in what areas and in what amount?

2. To what extent is the TA available commensurate with grantee (and fellows’) needs?

3. In what ways does the TA reflect and support major federal policy directions?

4. Report to Congress

1. How are the goals and progress of the Minority Fellowship Program communicated in SAMHSA’s Report to Congress?

Module D: Implementation of the MFP at Grantee organizations

1. Overview: Program Implementation

1. How do the grantees use the MFP resources from their SAMHSA grant?

2. Are there particular challenges faced by each of the grantees as they implement their Minority Fellowship Programs?

2. Recruit Fellows

1. What recruiting sources/mechanisms do grantees use to recruit fellows?

1. Which, if any, minority-specific venues do grantees use to recruit fellows?

2. How have the grantees sustained their ability to recruit new cohorts?

2. Select Fellows

1. What policies do grantees have for determining the eligibility of applicants?

2. How many fellows are funded each year by each of the grantees?

3. Do grantees meet their goals for selecting fellows?

4. What processes do the Advisory/Selection Committees of each grantee organization use to select fellows?

5. What processes do the Committees use for reviewing returning fellows in subsequent years?

6. How do the grantee programs monitor the progress of fellows, both at the end of the semester/year and on an ongoing basis?

3. Support and cultivate development of Fellows

1. How do grantees allocate their MFP dollars?

2. What formal supports do the grantees provide the fellows during the academic year?

3. What informal supports do the grantees provide the fellows? (E.g. on-line chats or ongoing mentoring relationships)

4. How have the grantees sustained their ability to mentor new cohorts?

   1. How many former fellows have mentored fellows?

5. What reasons do fellows provide for not completing the program?

5. Assess each grantee program

1. What evaluation activities and processes have the grantees used to assess the performance of their own program?

2. How is the information used to improve program performance?

3. How is information reported to various stakeholders (current fellows; former fellows; Advisory Board; members of the profession; SAMHSA; others interested in ethnic minority leadership in mental health and substance abuse)?

6. Report to Advisory Board

1. Who is on the Advisory Board?

2. How often does the Advisory Board meet?

3. What is the role of the Advisory Board for the grantee’s Minority Fellowship Program?

4. What information is reported to the grantee’s Advisory Board?

5. How is the information used by the Board? (Are Advisory Board members apprised of decisions; consulted about programmatic decisions; actively involved in making decisions)?

Module E: Proximal Outcomes

1. Develop new cohorts of fellows

1. How many applications do the grantees receive each year?

2. How many fellows are accepted each year?

3. In what ways has the cohort of fellows changed over the years in which SAMHSA has operated the program?

4. In training/mentoring students and mental health/substance abuse professionals outside of the MFP, what influence have former fellows had?

1. How many former fellows have encouraged students/professionals to pursue careers in treating ethnic minorities for mental health and substance abuse issues?

2. How many former fellows encouraged students/professionals to pursue careers in researching ethnic minority mental health and substance abuse?

2. Bolster the presence of ethnic/racial minority professionals; diversify and institutionalize leadership

1. What organizational structures or events have the grantee organizations established to facilitate job searches and professional development of ethnic/racial minorities into the mental health and substance abuse professions?

2. After leaving the MFP, in what professional contexts have former fellows worked?

3. In which professional associations have former fellows participated and in what capacity?

4. Have former fellows supervised clinicians serving minority communities?

3. Increase service and professional contributions of the MFP

1. What organizational structures or events have the grantee organizations established to increase the contributions of ethnic and racial minorities to mental health and substance abuse services?

2. How many former fellows have been employed by entities which focus their services on ethnic and racial minorities?

Surveys and interview protocols have been designed to assess these evaluation questions. The majority of the on-line survey items are pre-coded. (Respondents will be given the opportunity to write clarifying comments regarding their responses.) There are no correct or anticipated answers to questions.

There are two basic types of analyses to be conducted in this evaluation, using combined program data or grantee-specific data, each of which will use descriptive as well as inferential measures.

Combined MFP program data: The evaluation will use descriptive statistics such as frequencies, and means to report (1) the number and types of processes and activities established by SAMHSA and by the grantees to implement the MFP; (2) the percentage of Fellows (broken down by race/ethnicity, gender, and year) entering the program who have completed the program; (3) mean scores on attitudinal items assessing participants’ views about how well the SAMHSA MFP is performing; and (4) the degree to which program goals are being achieved.

Given the small sample sizes of some sub-groups (e.g. Program Directors and SAMHSA staff), some analyses will be descriptive and further generalizations beyond the data will not be possible.

However, data from the larger sub-groups (e.g. current and former Fellows) will be used to conduct inferential analyses. Such analyses will be conducted to examine a) differences between the experiences and assessments of different stakeholders (e.g. Alumni and current students); and b) differences in the program over time. (See Section B for a discussion of the sampling plan.)

The surveys and interview protocols have been constructed to include questions relevant to each of the respective stakeholder groups named above, with similar core questions asked across all groups. Therefore differences across groups may be assessed using either t-tests (for two groups) or Analysis of Variance, or ANOVA (for more than two groups). Other bivariate and multivariate relationships will be examined using correlations and tabular measures such as Chi-Square.

For example, to determine if Fellows from different cohorts have had access to different resources within the MFP, comparisons across cohorts will be made on measures such as “How many courses did you take at your university/ training program that were directly related to minority mental health and/or substance abuse. Other analyses will examine the number and types of publications and professional positions secured by Fellows, taking into account the year the Fellow completed training/defended their dissertation (and recognizing that more mature professionals will have had greater opportunities to secure these accomplishments.)

Grantee Specific Data:

All of the descriptive measures reported for the MFP program overall will also be reported separately for each of the grantees. That is, the following will be reported for the ANA, ApA, APA, and CSWE: (1) the number and types of processes and activities established by SAMHSA and by the grantees to implement the MFP; (2) the percentage of Fellows (broken down by race/ethnicity, gender, and year) entering the program who have completed the program; (3) mean scores on attitudinal items assessing participants’ views about how well the SAMHSA MFP is performing; (4) the degree to which program goals are being achieved. In addition, differences in stakeholder report/experience by each of the grantees also will be assessed as outlined in sample Tables 4 and 5.

Table 4: Outcome Data by Grantee (percent)

	ANA	ApA	APA	CWE	Totals
Fellows
Percent completing degree/residency within 5 years of entering
Percent completing degree/residency within 7 years of entering
Percent completing degree/residency within 10 years of entering

Table 5: Perceptions/Attitudes of Stakeholders by Grantee (means)

	ANA	ApA	APA	CWE	Totals
Advisory Committee Members
On a scale of 1 to 5, with 1 being very unhelpful and 5 being very helpful, how helpful would you say that your interaction with the ANA/APA/ApA/CSWE MFP office has been?

Although grantee-specific data will be reported in the core measures broken down by grantee (as noted above), there are other data which are grantee specific. For example, each grantee has used different structured learning opportunities than the others to prepare their Fellows for future professional responsibilities. Data such as these will be reported separately for each grantee.

Because of differences in program size, organization and culture, we expect to see differences among grantees. Furthermore, because respondents have purposely been selected with different perspectives, we expect to see differences due to respondent type as well. Therefore, appropriate inferential analyses will be used (ANOVA, t-test, Chi-Square, correlations or other measures of association), depending on the number of categories and level of data. A sample table (Table 6) demonstrates a cross-tabulation of responses, comparing means and standard deviations by type of respondent.

Table 6: Sample Table Shell

Thinking about [the goals for the MFP], how much progress would you say the SAMHSA MFP has made in its contribution toward achieving those goals?

1 = no influence on….

2 = very little influence on…

3 = some influence on…

4 = a fair amount of influence on…

5 = a great deal of influence on…

(Table to report ANOVA results: mean and standard deviation)

	Former Fellows	Current Fellows	Advisory Committee Member
Training/mentoring ethnic/racial minority students and professionals in mental health/substance abuse treatment
Increasing the number of ethnic/racial minority professionals in mental health/substance abuse treatment
Increasing diversity in mental health/substance abuse leadership
Increasing professional contributions in mental health/substance abuse treatment for minority populations
Increasing institutional involvement of ethnic/racial minority professionals in the areas of mental health and substance abuse treatment
Increasing mental health and substance abuse services to minority communities

17. Display of OMB Expiration Date

The expiration date for OMB approval will be displayed.

18. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submission.

B. Collection of Information Employing Statistical Methods.

1. Respondent Universe and Sampling Methods

Since 1992, over 570 doctoral students and psychiatric residents have been supported through SAMHSA’s MFP. Up until FY2007, the program has been operated through 4 grantee organizations (the ANA, ApA, APA, and CSWE). All four grantees will be included in the data collection and therefore the entire universe of SAMHSA grantees from 1992-2007 will be included. Different groups of stakeholders, however, will be selected differently as shown below:

Stakeholder Group	Selection Method: Universe or type of data collection	Estimated Number
Current SAMHSA MFP Fellows Survey	Universe	100
SAMHSA MFP Alumni (Former Fellows) Survey	Universe	850
MFP Selection and Advisory Committees Survey	Universe: All living current and former members holding the position for at least 1 full year (estimate: 40 participants)	40
Current and former MFP Program Directors or Senior Staff (from the grantee organizations) Interview Protocol	Universe of those currently alive	8
Grantee host organization Interview Protocol	2 members per grantee; Purposive sample (not in the MFP program but either having direct supervisory responsibility for overseeing or being most knowledgeable about the MFP program in the grantee association )	8

Sometimes, collecting new information about the full population is more appropriate than sampling, for both methodological and pragmatic reasons (Henry, 1990, Fink, 1995). For purposes of the MFP evaluation collecting new information from an entire population is useful for the following reasons (Fink, 1995):

 

1.   The size of the population of interest is so small that it is feasible to collect information from all members of that population.  The numbers served by the MFP program over the years has been small.  By sampling all we hope to obtain a very diverse set of responses over programs and time period.

 

2.   Collecting information from all members of the population may be perceived by stakeholders as more legitimate or credible.  Funders at SAMHSA and the grantee level are interested in hearing from all members in order to have as much information for program improvement as possible.

 

3.   Information on the entire population is readily available (e.g., automated information on critical incidents, automated data on claims and plans of care to assess degree to which authorized services are actually delivered, simple survey responses that do not place a large burden on respondents).  We have developed an online survey tool which makes it easy for respondents to fill out the form and provides automated skips for questions that are not applicable based on previous responses.  This minimizes the burden and makes the information readily available. 

 

4.   Sampling always involves some amount of error, so if you have the information you need on the entire population and it is readily available, it makes sense to use it rather than to sample. 

For the following four reasons above and the desire to obtain as much quantitative and qualitative data on the program as possible for program improvement we have decided to sample the universe, with the support of SAMHSA Project Officers. For this reason, power analyses have not been included since they are not necessary when the entire universe is included in the study.

 

2. Information Collection Procedures

Although one of the guiding principles in this evaluation of the MFP is that burden should be minimized, additional primary data collection is necessary. Since research has demonstrated that survey efforts are most effective when a knowledgeable and respected entity invites participation, all potential participants in this study (except the SAMHSA staff and the MFP Program Directors who already are aware of the evaluation) will receive a letter of invitation from the relevant grantee organization informing them about the upcoming study. The SAMHSA contractor will coordinate this effort and ensure that invitations are distributed. The mode of administration (on-line survey; or telephone interview) will be identified. For those who do not have access to the Internet, a toll-free “800” number will be available to request a hardcopy survey.

While mailed and electronic surveys are least intrusive to the respondent, anonymous, standardized, and relatively cost-efficient (in terms of time and money) to administer, they also tend to have the lowest response rates. Nonetheless, questionnaires/surveys will be the most effective strategy where there are sufficiently large numbers of respondents for which adequate response rates can be achieved (with appropriate follow-ups). Furthermore, since many stakeholders are familiar with and have access to the Internet, we will use on-line versions of surveys. Therefore, such surveys have been proposed where there are sufficiently large numbers of respondents which preclude the use of more labor-intensive strategies and in which an adequate response rate is likely to be achieved (with appropriate follow-ups): Current Fellows; MFP Alumni; and the Selection /Advisory Committees.

E-mail addresses will be provided by the MFP grantees and the invitation will include a link for recipients to use in answering an online survey. A reminder will be sent electronically (with the link to the survey again included) approximately 3 weeks after the initial e-mail. Three weeks after this first reminder, hardcopy questionnaires will be mailed out to non-responders. No subsequent reminders are planned.

In contrast, in-person (telephone) interviews have been identified for groups where the sample size is relatively small, telephone conferences may be a routine part of business, more complex or less structured/more process-oriented information will be gathered, and anonymity may not be as great a concern (such as SAMHSA program staff; the MFP Program Directors; and officials in the grantee host organizations).

Existing data sources were previously identified which were used to provide a report to SAMHSA on the following indicators:

1. The number of MFP Fellows completing their program/degree;

2. The number of MFP Fellows not completing their degree and terminating from their program;

3. The demographic profile (Gender and Race) of Fellows admitted into the MFP each year;

4. Current and past employment experiences of MFP alumni;

5. The number of MFP Fellows who have completed their program and are working in the public mental health system;

6. Populations MFP Fellows (alumni) currently are serving;

7. Average direct costs associated with training MFP Fellows; and

8. Specific measures of self-assessment selected by each of the professional associations to measure their program’s success.

In the course of analyzing data for the earlier report, the determination was made that existing data were insufficient to report on item 4, 5, and 6 above. The current evaluation has been designed to supplement, not duplicate that effort. No existing data source provides the information to be gathered in this effort.

3. Methods to Maximize Response Rates

Multiple strategies will be used to maximize response rates and minimize burden on respondents depending on the target respondent: a) use of an online survey tool; and b) in person phone calls.

Online surveys: The contractor’s IT staff will use web-based interface software to provide the respondents with a simple interface to transform a paper & pencil survey into a web-based survey. The software allows for multiple question types (yes/no, multiple choice, write-in responses, rating scales, etc.) and for skip patterns, also known as branching. The sites load very quickly and will be designed to be very user-friendly for respondents.

Respondents will receive an email requesting their participation in the survey and offer them the choice of completing the survey online or provide instructions for them to request a paper survey. The email will include a link to the website where the survey can be completed.

For those who request a paper and pencil survey, a hardcopy will be sent through the U.S. mail.

For those for whom we do not have an email address or do not respond to the initial email, we will attempt to contact them through telephone and mail and offer them a choice of on-line or hardcopy. The Research Assistant working on the evaluation will follow up with non-respondents; two additional attempts (beyond the initial contact) will be made to elicit participation.

Given the central role the MFP played in funding their professional training, relatively high response rates are expected for each survey including:

• 85% for Current Fellows;

• 70% for MFP Alumni, and

• 80% for Advisory/Selection Committee Members.

Telephone Survey: Multiple attempts will be made to schedule the telephone interviews with SAMHSA staff, MFP Project Directors in the grantee programs; and with officials in the grantee host organizations. Although these individuals have substantial interest in the MFP program, they have extremely busy schedules. Therefore, interviews will be scheduled when these participants are available (including evenings and weekends). Two senior interviewers will participate in each interview and take turns either leading the interview or taking notes. With permission of the interviewee, interviews will also be tape-recorded to capture details; all audiotapes will be destroyed at the completion of data analysis. Should a participant not wish to be tape-recorded, no recording will be made.

A 90% response rate from these groups is expected given their investment and participation in the MFP.

4. Analysis Procedures

Univariate, bivariate, and multivariate analyses of all variables (moderating variables, intervention components, mediating variables, and outcome components) will be examined overall, and by grantee Bivariate and multivariate analyses may include comparisons of variables and demographics within and between grantees to identify important covariates and confounding variables prior to examining the primary hypotheses.

5. Tests of Procedures

All survey instruments were tested. The two advisory panels on this project (the Advisory Panel and Subject Matter Experts, See Section A, 8. Consultation Outside the Agency) first assisted in framing the research questions which are guiding the evaluation. Members of the two panels then reviewed the survey and interview questions developed by the contractor based on these research questions. Members of the two panels were asked to assess whether or not the interviews and survey questions a) were appropriate for their professional discipline; b) adequately reflected the way in which the grants were administered in their program; and c) would adequately capture information to address the research questions. Only minor changes were suggested, primarily consisting of word choice to clarify intent or recommendations to add questions to further explore an issue.

6. Statistical Consultants

SAMHSA: Phone:

Antoine Smith, M.P.H. SAMHSA Government Project Officer 240- 276-2006

Human Services Research Institute will have overall responsibility for implementation and execution for the evaluation and for the preparation of all reports.

Contractors: Phone:

Virginia Mulkern, Ph.D. Human Services Research Institute 617-876-0426

Terry Camacho-Gonsalves, Ph.D. Human Services Research Institute 617-844-2504

David Hughes, M.A.. Human Services Research Institute 617-844-2527

Bibliography

Henry, G.T. (1990)  Practical Sampling, Applied Social Research Methods Series, Volume 21

Thousand Oaks, CA: SAGE Publications.

 

Fink, A. (1995)  How to Sample in Survey.  Thousand Oaks, CA: SAGE Publications.

List of Attachments:

SURVEY AND INTERVIEW PROTOCOLS

• Attachment A Survey: Current SAMHSA MFP Fellows

• Attachment B Survey: MFP Alumni

• Attachment C Survey: Current and former members of Selection and Advisory Committees

• Attachment D Interview protocol: Current and former SAMHSA MFP Staff and other SAMHSA officials involved in the MFP

• Attachment E Interview protocol: Current and former MFP Program Directors/Senior Staff in each of the four grantee programs

• Attachment F Interview protocol: Staff in each of the grantee’s host organizations (i.e. staff in the ANA, APA, ApA, and CSWE)

OTHER INFORMATION AND RESOURCES:

• Attachment G Example survey page displaying the OMB control number, expiration date, and burden statement

• Attachment H Membership Lists: Advisory Panel and Subject Matter Experts

• Attachment I Consent Form

• Attachment J Telephone Script

• Attachment K Letter of Invitation: Email

• Attachment L Letter of Invitation: US mail

3