Ss_part A

SUPPORTING STATEMENT

PREVENTIVE MEDICINE FELLOWSHIP RESIDENCY PROGRAM EVALUATION

Part A

Elinor Greene, Ph.D.

Career Development Division

Office of Workforce and Career Development

Centers for Disease Control and Prevention

2400 Century Center Parkway, NE, MS E92

Atlanta, Georgia 30333

Email: [email protected]

Phone: (404) 498-6156

PREVENTIVE MEDICINE FELLOWSHIP RESIDENCY PROGRAM EVALUATION

This supporting statement is for a 3-year clearance for the collection of data using an electronic survey to evaluate the effectiveness of CDC’s Preventive Medicine Residency Fellowship under the 42 USC, Section 301(241) of the Public Health Service Act (Attachment 1). CDC’s Mission is to promote health and quality of life by preventing and controlling disease, injury, and disability. Preventive medicine is a specialized field of medical practice that works with large populations to promote good health; to prevent disease, injury and disability; and to facilitate early diagnosis and treatment of illness. The mission of CDC's Preventive Medicine Residency and Fellowship (PMR/F) is to train public health and preventive medicine leaders, and maintain leadership in the field of preventive medicine training. The last formal evaluation of this program was conducted over 16 years ago, and the structure and design of the program changed in 1996. The proposed evaluation will focus on achievement of targeted outcomes by using a reference group matched on all critical variables with the exception of PMR/F training. The study group will consist of all eligible PMR/F alumni who are also Epidemic Intelligence Service (EIS) alumni who entered their practicum year on or between 1996 and 2006. The reference group will consist of twice as many matched and randomly selected EIS alumni who were eligible to apply to and enter the PMR/F practicum year between 1996 and 2006, but did not apply. In addition, PMR/F alumni will be asked about their leadership activities, the usefulness of their PMR/F training activities, and any suggested additional activities that they might recommend for the PMR/F curriculum. The results of the evaluation will be used to provide recommendations for continuing the program as is, or for improvements to curriculum or other program elements.

A. JUSTIFICATION

A.1. Circumstances Making the Collection of Information Necessary

CDC’s Mission is to promote health and quality of life by preventing and controlling disease, injury, and disability. CDC’s Office of Workforce Development’s Career Development Division is home to CDC’s Preventive Medicine Residency, one of 12 scientific residencies and fellowships whose focus is to support CDC’s mission and vision. The career Development Division’s mission is to be a leader in the creation and execution of training and service programs that develop competent public health professionals.

Preventive medicine is a specialized field of medical practice that works with large populations to promote good health; to prevent disease, injury and disability; and to facilitate early diagnosis and treatment of illness. It is unique from most areas of medicine because its central focus is population health. Despite the nation's growing need for leaders in prevention medicine, numerous studies have demonstrated an increasing shortage of preventive medicine-trained professionals, and that shortage is projected to continue (American College of Preventive Medicine; Council on Graduate Medical Education, Institute of Medicine). Preventive efforts are perceived as being limited to industrialized countries; however, preventive medicine is widely practiced in third-world nations, for example, leading vaccination and water supply and sewage disposal programs. The growing shortage of well-trained leaders and practitioners in public health and preventive medicine is felt around the world. The specialty will benefit from attracting new residents and future leaders in public health and preventive medicine to enter rewarding programs that fill positions with highly qualified candidates, and expand the specialty into new medical leadership roles (Ducatman, et al., 2005).

The mission of CDC's Preventive Medicine Residency and Fellowship (PMR/F) is to (1) train public health and preventive medicine leaders, and (2) maintain leadership in the field of preventive medicine training. CDC's PMR/F has been training physicians in the residency since 1972 and veterinarians in the fellowship since 1983. The program consists of a one-year practicum, based either at the CDC or at a state or local health department, and sponsorship for a Master of Public Health degree for qualified physician applicants before the practicum year. All alumni of the program have completed CDC’s Epidemic Intelligence Service, a two-year program in applied epidemiology. PMR/F provides its residents and fellows with training and experience in leadership, management, evaluation, and program development and evaluation.

The field of Preventive Medicine has grown tremendously during the past few years although little has been done to evaluate CDC’s PMR/F and its targeted outcomes. CDC proposes to evaluate the PMR/F to determine: (1) How well PMR/F is fulfilling its mission to train competent public health leaders and (2) the effectiveness of the PMR/F competency-based curriculum toward developing public health leaders.

The proposed program evaluation will focus on the practicum year and attainment of targeted outcomes in the areas of leadership, contribution to public health and preventive medicine, and career development. The evaluation will include a study group consisting of all eligible PMR/F alumni who entered their practicum years on or between 1996 and 2006 and a matched and randomly selected reference group who did not apply, and had they applied, would have entered their practicum year the same years. The structure and design of the program changed in 1996; therefore, the evaluation will be limited to individuals who entered or were eligible to apply and enter on or after 1996.

The CDC Preventive Medicine Residency (PMR) is accredited by the Accreditation Council for Graduate Medical Education (ACGME) as a 12-month program meeting the Practicum Year training requirements for certification by the American Board of Preventive Medicine (ABPM) in Public Health and General Preventive Medicine. The Preventive Medicine Residency Fellowship (PMR/F) prepares physicians and veterinarians for future leadership roles in public health at federal, state, and local levels. Physicians apply to the residency and veterinarians apply to the fellowship. All programmatic elements of the residency and fellowship are the same, including competencies and curriculum. Physicians and veterinarians who apply to enter the PMR/F practicum year must meet specific eligibility requirements to be considered for selection. See Attachment 4 for details of eligibility requirements.

Physicians who are accepted and do not have a Masters in Public Health (MPH) or equivalent degree receive support for the MPH and complete their academic year (MPH) immediately before they enter their practicum year. To receive support, physicians must be officers in the U.S. Public Health Service Commissioned Corps. Physicians who successfully complete their practicum year are eligible to sit for the American Board of Preventive Medicine (ABPM) certification exam.

Veterinarians who are accepted must have an MPH or an equivalent degree; consequently, they do not receive support for their academic year. Veterinarians who successfully complete their practicum year are generally eligible to sit for the American College of Veterinary Preventive Medicine (ACVPM) certification exam; however, their application fees are not funded by PMR/F.

PMR/F uses logic model methodology for both planning and evaluation. PMR/F’s logic model was developed with input from its Graduate Medical Education Committee (GMEC), Residency Advisory Committee (RAC), and Working Group including PMR/F alumni. The fellow-focused logic model depicts the residents/ fellows progression through their practicum year and beyond. It includes inputs to the fellowship, the educational curriculum, outputs, and targeted short-term, intermediate, and long-term outcomes. PMR/F’s programmatic logic model portrays the key activities required to develop, manage, and evaluate the residency/fellowship. The outputs of the programmatic logic model are inputs to the fellow-focused logic model. PMR/F’s programmatic and fellow-focused logic models are seen in Attachment 5.

The PMR/F program evaluation will focus on short-term (1-3 years), intermediate (4-7 years), and long-term (8 years or longer) outcomes indicated on the fellow-focused logic model. Each level of outcome includes targets in the following three categories: leadership, career development, and contribution to public health. Alumni who entered their practicum year between 2004 and 2006 might be expected to have achieved at least the short-term outcomes, alumni who entered their practicum year between 2000 and 2003 might be expected to have achieved at least the intermediate outcomes, and alumni who entered their practicum year between 1996 and 1999 might be expected to have achieved at least the long-term outcomes. The outcomes are seen in Table 1.

Table 1. PMR/F Logic Model Outcomes

4.0 Short-Term Outcomes (1-3 years)	5.0 Intermediate Outcomes (4-7 years)	6.0 Long-Term Outcomes (8 years or longer)
4.1 Leadership 1. Alumni demonstrate leadership within PH/PM organizations 2. Alumni participate in the analyses, development, implementation, and evaluation of PH/PM programs and policies 3. Alumni apply systems-based approach in addressing health problems 4. Alumni apply population-based principles in addressing health problems	5.1 Leadership 1. Alumni in leadership roles across PH/PM organizations 2. Alumni participate in agenda setting and vision and mission development 3. Alumni analyze, develop, implement, and evaluate PH/PM programs, policies and performance standards	6.1 Leadership 1. Alumni are leaders in PH/PM on a national and international level 2. Alumni set agendas, create visions, and define missions 3. Alumni oversee the analyses, development, implementation, and evaluation of PH/PM programs, policies and performance standards of National and global scope
4.2 Career Development 1. PMR/F training was relevant and prepared alumni for current PH/PM professional activities 2. Alumni are working in PH/PM 3. Alumni pass their board exam within two years of graduation	5.2 Career Development 1. PMR/F training was relevant and prepared alumni for current PH/PM professional activities	6.2 Career Development 1. PMR/F training was relevant and prepared alumni for current PH/PM professional activities
4.3 Contribution to PH/PM 1. Alumni contribute to the PH/PM body of knowledge	5.3 Contribution to PH/PM 1. Alumni contribute to and use the PH/PM body of knowledge for addressing health problems across population groups	6.3 Contribution to PH/PM 1. Alumni contribute to the use of, and influence the interpretation of the PH/PM body of knowledge

A.2. Purpose and Use of the Information

The outcome of the evaluation may include recommendations for modifying curriculum, pre-requisite qualifications, mentoring, or other program elements. The survey will be anonymous and confidential and the data will be shared with those with a need to know.

A.3. Use of Improved Information Technology and Burden Reduction

The survey will be electronic, delivered by Mr.Interview^TM, part of SPSS Dimensions. Mr. Interview^TM is Certificate and Accreditation (C&A) approved, and the link to the survey will be sent to respondents via email. The information will be collected via email and stored in a secure database. Only selected respondents will receive the survey; it will be inaccessible to and will not burden the general public. The proposed data collection instrument is found in Attachment 3.

A.4. Efforts to Identify Duplication and Use of Similar Information

PMR/F alumni and a comparable reference group consisting of EIS alumni who did not apply to PMR/F will receive and complete the survey once. This information is neither readily accessible nor available from any other source. The data will be used to evaluate PMR/F and to improve the program if necessary. There will be no duplicate data collection.

A.5. Impact on Small Businesses or Other Small Entities

Physicians and veterinarians who participated in EIS are the target audience of this data collection. The information requested of them is voluntary.

A.6. Consequences of Collecting the Information less Frequently

There are no legal obstacles to reduce the burden.

A.7. Special Circumstances Relating to the Guidelines of CFR 1320.5

This request fully complies with the regulation.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The agency’s notice of proposed data collection was printed in the Federal Register on March 14, 2007 (Volume 72, Number 49, pages 11887-11888 in Attachment 2). One public comment was received in response to the notice (Attachment 6). No changes were made to the proposed project based on this response, as the public comment did not relate to the utility and scope as proposed.

CDC enlisted BearingPoint, Inc., a consulting firm with expertise in program evaluation to design and conduct the evaluation. BearingPoint’s headquarters are located at 1676 International Drive, McLean, VA. 22102 and its local office is located at 115 Perimeter Center Place. Suite 380. Atlanta, GA 30346. David Friedman is the point of contact at BearingPoint and can be reached at [email protected].

Within CDC, PMR/F staff, the CDD director and chief science officer, the EIS staff statistician, the Graduate Medical Education Committee (GMEC), Resident Advisory Committee (RAC), and working group were consulted to determine the need for and the design of the program evaluation.

A.9. Explanation of any Payments or Gifts to Respondents

There are no payments or gifts to respondents.

A.10. Assurance of Confidentiality Provided to Respondents

This submission has been reviewed by ICRO staff for Privacy Act applicability and it was determined that the Privacy Act does not apply. Data will be anonymous and no questions will be asked of a personal nature. The survey will reside on a server managed by CDC’s Information Technology Services Office (ITSO) under strict physical security. ITSO Data Center personnel will have access to the physical server. Registrant data will not be sold, rented or shared with third parties for their promotional use. All data will be maintained behind a strict firewall with security protection.

The surveys will be submitted through the Internet. Data will be stored on a secure Microsoft SQL Server database located behind the CDC firewall. Access to information on the CDC Microsoft SQL Server database is available only to CDD administrative personal and OWCD IT staff. Security provisions for data storage meet all requirements established by CDC’s Information Council Executive Committee (CICEC).

A.11. Justification for Sensitive Questions

No questions of a sensitive nature will be asked of respondents.

A.12. Estimates of Annualized Burden Hours and Costs

The only cost to the respondents is the time involved to enter their responses to the survey. For each of the 90 respondents, one response will require an average of 30 minutes to complete the survey. Annualized over three years, 30 responses each year will require an average of 30 minutes to complete, resulting in an annualized respondent burden of 16 hours.

A12-A. Estimates of Annualized Burden Hours

Table 2. Preventive Medicine Residency Fellowship Evaluation Respondent Burden*

Type of respondents	Number of respondents	Number of responses per respondent	Average burden per response	Total burden hours
Study Group Physicians	8	1	30/60	4
Reference Group Physicians	17	1	30/60	9
Study Group Veterinarians	2	1	30/60	1
Reference Group Veterinarians	3	1	30/60	2
Total	30			16

Total response burden annualized over three years.

The estimates of time are based on a pilot test conducted with 6 respondents. The average time to complete the survey for both the study group and reference group is 30 minutes.

Average time to complete survey in study group	30 minutes
Average time to complete survey in reference group	30 minutes

A-12B. Estimates of Annualized Cost Burden

The estimates of annualized cost burden for respondents were developed by first calculating the number of physicians and veterinarians who will be surveyed and next determining their average salary.

The following source was used to determine the average salaries of participants:

• US Department of Labor, Bureau of Labor Statistics

http://www.bls.gov/oes/current/oes_nat.htm

Table 3. Preventive Medicine Residency Fellowship Evaluation Respondent Cost

Type of respondents	Total Burden Hours	Average Hourly Wage Rate	Total Respondent Cost
Study Group Physicians	4	$72	$288
Reference Group Physicians	9	$72	$648
Study Group Veterinarians	1	$39	$39
Reference Group Veterinarians	2	$39	$78
Total	16		$1053.00

Total cost burden annualized over three years.

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers

There are no other capital or maintenance costs to respondents.

A.14. Annualized Cost to the Government

The annualized cost to the federal government for performance of this survey and analysis of the data is approximately $1,053.. There will be no fee for using Mr. Interview (CDC Certificate and Accreditation [C&A] approved) and SPSS. BearingPoint contractors will perform survey development, design, administration, data review, data analysis, and provision of recommendations.

A.15. Explanation for Program Changes or Adjustments

This is a new data collection.

A.16. Plans for Tabulation and Publication and Project Time Schedule

No publications or manuals are planned.

Timeline of Key Events following Receipt of OMB Clearance

Key Events	Timeline
Receive OMB clearance
Program survey into package	Three months after receipt of OMB clearance
Start information collection / Send survey links to respondents	Four months after receipt of OMB clearance
Send reminders to respondents/ Send survey links to respondents	Six months after sending survey
Send reminder to respondents/ Send survey links to respondents	1 month after sending survey
Send reminder to respondents/ / Send survey links to respondents	2 months after sending survey
Send reminder to respondents/ Send survey links to respondents	5 month after sending survey
Send reminder to respondents/ / Send survey links to respondents	8 months after sending survey
Send reminder to respondents/ Send survey links to respondents	12 months after sending survey
Send reminder to respondents/ Send survey links to respondents	18 months after sending survey
Analyze data /Develop recommendations write report	16-18 months after sending survey
Present recommendations to PMR/F Director and staff	18 – 24 months after sending survey
Make revisions and resubmit	26 – 28 months after sending survey

A.17. Reasons Display of OMB Expiration Date is Inappropriate

CDC is not requesting an exemption from displaying the expiration date.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to certification for Paperwork Reduction Act Submissions anticipated.

9