Backup of SS_ PART B.wbk

Supporting Statement, Part B December 27, 2007

SUPPORTING STATEMENT

PREVENTIVE MEDICINE FELLOWSHIP RESIDENCY PROGRAM EVALUATION

Part B

B. Collections of Information Employing Statistical Methods

Qualitative analyses, descriptive statistics, proportions, histograms, and a contingency table analysis will be used to compare the two groups and to make recommendations for improvements to the curriculum or other program elements.

B.1. Respondent Universe and Sampling Methods

The proposed program evaluation includes a study group and reference group. The study group consists of 30 PMR/F alumni who are also EIS alumni and entered their practicum year on or between 1996 and 2006. The reference group consists of 60 EIS alumni who were eligible to apply to PMR/F, did not apply, and could have entered the practicum year on or between 1996 and 2006. We propose to survey twice as many members in the reference group (60) as in the study group (30), to increase reliability. The hypotheses for the program evaluation are listed below:

1. EIS alumni who are also PMR/F alumni obtain leadership roles in fewer years than the reference group,

2. PMR/F training is relevant and prepares its alumni for current PH/PM professional activities.

Of the 30 PMR/F alumni who will be surveyed, 25 are U.S. physicians and 5 are U.S. veterinarians. They will be assigned to one of three pools based on the year they could have achieved logic model outcomes. Pool 1 will consist of alumni who entered the practicum year between 2004 and 2006 and could have achieved short-term outcomes. Pool 2 will consist of alumni who entered the practicum year between 2000 and 2003 and could have achieved intermediate outcomes. Pool 3 will consist of alumni who entered the practicum year between 1996 and 1999 and could have achieved long-term outcomes. Study and reference group assignments and pools are found in Tables 4 and 5.

Table 4. Total Number of Study Group (SG) Members in Each Pool

Pool	Year SG members entered practicum Year	Target logic model outcomes	Projected years to achieve target outcomes after graduation	Total members in SG
1	2004-2006	Short-term	1 – 3 years	7
2	2000- 2003	Intermediate	4 – 7 years	12
3	1996-1999	Long-term	8 years or longer	11
			Total:	30

Table 5. Total Number of Reference Group (RG) Members in Each Pool

Pool	Year RG members could have entered practicum Year	Target logic model outcomes	Projected years to achieve target outcomes after projected graduation year	Total members in RG
1	2004-2006	Short-term	1 – 3 years	14
2	2000- 2003	Intermediate	4 – 7 years	24
3	1996-1999	Long-term	8 years or longer	22
			Total:	60

The proposed reference group will be identical to the study group in all influencing factors with one exception; participation in PMR/F. The reference group will be matched to the study group on the following variables:

1) EIS alumni who met all PMR/F application requirements

2) Professional category of participants (percentage of physicians and veterinarians in each pool)

4) Status of MPH for physicians (percentage of physicians without MPHs matched to percentage PMR/F alumni who sought support for the MPH in each pool)

5) The year reference group members would have entered the practicum year had they applied (Years of work experience between EIS and PMR/F academic year)

The final reference group will be randomly selected from each pool using a randomized process. The ratio of reference group members to study group members between 1996 and 2006 is approximately 4:1.

Table 6. Composition of Study Group

Pool	# Veterin- arians in SG	# Physicians who did NOT receive support for MPH and worked 0-3 years before practicum year	# Physicians who DID receive support for MPH and worked 0-3 years before academic year	# Physicians who DID receive support for MPH and worked 4-7 years before academic year
1	1	4	1	1
2	2	3	6	1
3	2	4	5	0
Total:	5	11	12	2

Table 7. Composition of Reference Group

Pool	# Veterin- arians in SG	# Physicians who had an MPH and worked 0-3 years following EIS before they could have entered their practicum year	# Physicians who did not have an MPH and worked 0-3 years following EIS before they could have entered their academic year	# Physicians who did not have an MPH and worked 4-7 years following EIS before they could have entered their academic year
1	2	8	2	2
2	4	6	12	2
3	4	8	10	0
Total:	10	22	24	4

B.2. Procedures for the Collection of Information

Respondents will be sent the link to the survey and they will take the survey on-line.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

Although participation in the survey is voluntary, every effort will be made to increase participation. Participants will receive frequent reminders by email to complete the survey and they will be told of the tremendous importance of their participation.

B.4. Tests of Procedures or Methods to be Undertaken

The analysis of the survey data will be organized to address the evaluation goals and assist the PMR/F program to have information to assist with program descriptions, recruitment and addressing curriculum needs. Four areas of data analysis will be conducted:

1. Summaries of leadership activities and contributions to public health accomplished by PMR/F alumni;

2. summaries of the advancement of PMR/F alumni and reference group to advanced leadership activities;

3. summaries of important PMR/F curriculum activities that alumni report provided the greatest contribution to their job activities and career progression; and

4. summaries of general recommendations provided by PMR/F alumni for the program leadership.

Summaries of leadership activities and contributions to public health accomplished by PMR/F alumni

The PMR/F is interested in knowing the number and proportion of alumni who remain in public health practice and who progress in their careers to advanced leadership activities. Previously, the only information available to the program was knowledge of the first job taken following completing the program. The number and proportion of alumni reporting that they work in job categories (government, academic, private practice, etc.), the level of responsibility that they perform and the proportion of time spent in public health or population health practice, and medical board specialty will be summarized using tallies, proportional statistics and histograms. An important analytic component will be to summarize the number and proportion of alumni who have attained board certification in preventive medicine or veterinary preventive medicine.

Summaries of the advancement of PMR/F alumni and reference group to advanced leadership activities

As a public health leadership program, the PMR/F is very interested in determining whether alumni of the program progress more rapidly to more advanced career leadership activities than those who have not had similar training. The initial analysis of data obtained from the survey will be to summarize leadership activities. These activities include the role they play in their organization of employment as well as their leadership contributions to the profession of public health. Descriptive tabulations and narratives will be used to summarize this information. Data analysis to determine whether alumni of the PMR/F program progress more rapidly than similar reference counterparts will be conducted using very basic comparisons. Both PMR/F alumni and the reference alumni of the Epidemic Intelligence Service (EIS Officers) will be assessed using the survey data to determine whether they are functioning in early-career, mid-career, or advanced leadership skills using the definitions of each of these activities contained in the program evaluation logic model. A contingency table analysis will be conducted using a basic chi-square analysis to determine if the PMR/F alumni are progressing to more advanced leadership activities. Expected numbers in each cell, for each of the PMR/F and EIS alumni, will be based upon the duration since completing the respective programs. The observed level of leadership will be compared with the expected using a contingency analysis as outlined below:

Alumni	Categories of Career
	Early Career (1-3 years)	Mid-Career (4-7 years)	Advanced Career (8 or more years)	Total
PMR/F & EIS	Observed vs. Expected	"	"
EIS only	"	"	"
Total

Summaries of important PMR/F curriculum activities that alumni report provided the greatest contribution to their job activities and career progression

Each major training activity is identified in the survey and each PMR/F survey respondent will be asked to report the importance of that activity for their career development. These responses will reported as a modified “Likert” scale. The responses will be tallied and the proportion of each type of response will be reported. In addition, the responses will be dichotomized into “strongly agree” plus “agree” responses and those with neutral or negative responses. The number and proportion of responses in these categories will also be reported. Histograms or bar charts may be used to provide a visual representation of the responses. Leadership skills reportedly learned through the EIS program will be similarly reported. Since no two activities between the programs are exactly alike, direct analytic comparisons cannot be conducted. Narrative descriptions of similarities will be provided to describe to the PMR/F leadership how leadership skills may be obtained through alternative teaching activities.

Summaries of general recommendations provided by PMR/F alumni for the program leadership

PMR/F alumni will be provided open-ended opportunities to offer suggestions for improvements to the PMR/F. These questions will generally be suggestions for other skills of importance that should be taught. These suggestions will be listed, combining suggestions of similar intent. A narrative summary will be provided for these suggestions.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The Designated BearingPoint contractors will collect and/or analyze the data for completeness. They will review the surveys for completeness. The Epidemic Intelligence Service staff statistician has been consulted.

B.6. CDC Contacts

Elinor Greene, Ph.D.

Career Development Division

Office of Workforce and Career Development

Centers for Disease Control and Prevention

2400 Century Center Parkway, NE, MS E92

Atlanta, Georgia 30333

Email: [email protected]

Phone: (404) 498-6156

Attachments

Attachment 1 Public Health Service Act

Attachment 2: Federal Register on March 14, 2007 (Volume 72, Number 29, page 11887-11888)

Attachment 3: Proposed data to be collected through the PMR/F Evaluation Survey

Attachment 4: PMR/F Eligibility Requirements

Attachment 5: PMR/F Logic Model

5