Supporting Statement A

Supporting Statement

For OMB Information Collection Request

Part A

2/19/2014

Initial Assessment and Evaluation of Public Health Training Centers Programs

A. JUSTIFICATION

A.1. Circumstances Making the Collection of Information Necessary

Background

The Institute of Medicine’s The Future of Public Health asserted that the functions of governmental public health agencies (PHAs) in the United States are three-pronged: regular and systematic assessment of community health needs; development of public health policies to support use of the scientific knowledge base in public health programs and policies; and assurance that constituents receive necessary personal health services, including subsidization of direct provision of services (Institute of Medicine, 1988). Since the publication of this seminal report in 1988, the field of public health has used these three roles as the focus for defining the mission of PHAs. These three functions have become the foundation for defining 10 essential public health services, national public health performance standards, the operational definition of local public health departments, and public health accreditation (CDC, 2010; National Association of County and City Health Officials, 2005; Public Health Accreditation Board, 2012).

PHAs require a sufficient, well-trained workforce to meet the current and emerging challenges to the nation’s public health. This requirement is the basis upon which the Health Resources and Services Administration (HRSA) strives to improve access to health care services for those who are uninsured, isolated, or medically vulnerable. HRSA’s Public Health Training Center (PHTC) Program supports the needs of the public health workforce through training of new and existing public health professionals.

The PHTC Program’s budget remained relatively stable (between $4.5 and $5.5 million) from the program’s inception in 1999 to 2010. In 2010, the Patient Protection and Affordable Care Act’s Prevention and Public Health Fund (PPHF) provided additional funds so that the 2011 budget was $23.1 million. This increase in funding resulted in an increase in the number of PHTCs from 11 to the present number of 37. These PHTCs have cooperative agreements and are located in accredited schools of public health and other public or not-for-profit institutions. In addition to statutory requirements,¹ PHTCs themselves are programmatically required to provide competency-based training and educational programs based on the Core Competencies for Public Health Professionals outlined by the Council on Linkages Between Academia and Public Health Practice, supporting core public health functions and the Ten Essential Public Health Services (Council on Linkages Between Academia and Public Health Practice, 2010).

In order to meet these requirements, PHTCs conduct an array of activities that reach a range of public health professionals. PHTCs offer a mix of relatively brief, informal presentations that are designed to be viewed online; webinars which target public health professionals but may be accessed by others, such as policy makers or governmental officials who may also benefit from the resource; and in-person training opportunities and focus on creating linkages between public health and allied governmental and not-for-profit agencies and organizations. Some PHTCs offer certificate programs or other, more structured resources.

Despite the PHTC Program’s efforts to support the development of the public health workforce, the Association of Schools and Programs of Public Health (ASPPH) estimates that by 2020, the US will need 250,000 more public health professionals than currently projected (Association of Schools of Public Health, 2008). This figure highlights the need to increase the number of public health professionals as well as the competence of those within the workforce. Moreover, those within the workforce require the skills necessary to navigate the ever-changing public health system. Changes in financing due to implementation of the ACA, global health, and state-level fiscal challenges mean that the public health workforce requires not simply personnel, but those with the relevant skills.

Assessing the impact of PHTCs requires an approach that reflects a deep understanding of public health practice and a sophisticated approach that produces results that facilitates HRSA’s informed decision making. HRSA has contracted with RTI International (RTI) to provide the experience and expertise needed to meet these requirements.

RTI has developed a program evaluation methodology to collect data that will inform decisions made by HRSA regarding PHTCs. The program evaluation consists of a multi-mode approach in which qualitative and quantitative data will be collected and analyzed using a variety of methods. A brief overview of the full program evaluation methodology is represented in the exhibit below:

First an assessment of PHTC program data was completed to provide HRSA with a better understanding of PHTC program gaps in terms of evaluation. Next a design methodology was completed to guide the development of focus group questions and a survey of PHTC trainees. Focus groups were then conducted to gather in-depth, qualitative data about experiences with longer-term, intensive PHTC programs. A survey of PHTC trainees will be conducted to collect primarily quantitative data about their experiences with regard to relatively short courses offered by PHTCs. RTI will then conduct analyses of all these data and report findings to HRSA to assist in decisions regarding PHTC programs. This submission focuses on the methodology of the survey of PHTC trainees and the analyses to be conducted with the survey data.

To assist RTI in conducting the survey, we will be utilizing the TrainingFinder Real-time Affiliate Integrated Network Learning Management System (TRAIN LMS), a learning resource for public health professionals. TRAIN LMS is comprised of a national site with many state and local affiliate sites which are managed by public agencies, academic partners, and others. Through TRAIN LMS, PHTC trainees can search for onsite or distance learning courses, sign up to receive notices about specific courses, register for courses, create personal records of completed training courses, and provide feedback about courses they have completed. Through HRSA TRAIN LMS, course instructors can publicize their courses, manage online rosters, receive feedback regarding their courses, and post course materials and discussion topics. RTI will use the HRSA TRAIN LMS to assist in sampling the population of PHTC trainees and the survey administration.

The trainee survey will be conducted to collect specific data about individual PHTC courses. As described below, the survey will collect data on such things as the mode of administration of the course, the pertinence of the course material to the trainee, and overall evaluation criteria regarding the course. Not all PHTC courses are the same. For example, some courses are administered in-person while others are offered via the web. The topics of courses vary within and by PHTCs. However, the survey questions have been developed to capture data regarding all courses no matter the differences in course characteristics.

1.1 Privacy Impact Assessment

(i) Overview of Data Collection System

The survey has been developed by HRSA in conjunction with RTI. The questions were based on informal evaluations administered by PHTCs and were informed by the Kirkpatrick model². The sampling strategy will consist of sampling all participants of PHTC courses until an appropriate response rate is reached to achieve the statistical power necessary to present results that will be beneficial to HRSA and the PHTCs. This approach will be taken because little is known regarding data collection with this population. There is little evidence to anticipate expected response rates with this population (participants in courses registered in HRSA TRAIN LMS ). Participants will be selected from the universe of all participants in PHTC courses administered during a pre-determined time frame, specifically the summer of 2014. While contact information will be necessary to inform the participants of their selection in the survey, no identifying information will be collected during the evaluation, and that contact information will be kept separate from survey data.

2. Items of Information to be Collected

Information is collected in a one-time anonymous web survey (Attachment F). Questions are asked about basic demographics of sample members, satisfaction questions regarding the course taken, knowledge-based questions regarding items learned during the course/training, and applicability questions to determine how what was learned during the course does or may apply to future work. Data will be collected from some respondents shortly after the course has been completed or for other respondents up to 3 months after course completion, depending on when the course was taken. Respondents will not receive an incentive for participating in the study.

No personal identifiable information is collected during the survey. However, names and contact information will be provided to RTI International via the HRSA TRAIN LMS to contact sample participants. These data will be securely stored separately from any survey data.

A.2. Purpose and Use of Information Collection

The objectives of the national evaluation are to assess how the PHTC Program is:

1. meeting the needs of the public health workforce;

2. improving the public health workforce by strengthening the competence of the existing and future public health workforce; and

3. addressing shortages of the public health workforce.

The specific aims are to collect key evaluation data regarding the effectiveness of training courses conducted by PHTCs. Specifically, a survey will collect data to be used by HRSA and the PHTCs to improve training course materials, instruction, and applicability to the workforce. These data will assist in developing new programs and materials that will assist public health workers and PHTCs to improve the public health workforce.

2.1 Privacy Impact Assessment

Provide the following:

1. A description of how the information will be shared and for what purpose

The data will be used by HRSA and only shared with PHTCs and the public at an aggregate level. They will be used to provide feedback to HRSA and PHTCs regarding the effectiveness of specific training courses in order to improve those courses as they relate to job performance. The data may also be used to disseminate program outcomes to academic and lay audiences about the effectiveness of the PHTC program in developing and implementing need-based continuing education offerings.

2. A statement detailing the impact the proposed collection will have on the respondent’s privacy:

HRSA will have access to names and contact information of all those who register for a PHTC training course via the TRAIN system. However, HRSA will not know which participants will be selected into the sample nor will they know which participants complete the survey. No data collected in the survey will be linked to respondent information as the survey will be completed via the Web. The data in the TRAIN system will only be used to contact sampled participants. Only limited demographic information is requested (e.g., race, highest level of education).

All data are maintained in a secure manner throughout the data collection and data processing phases. Only RTI International personnel who are conducting the study and have a study-specific need to know have access to the temporary information that could potentially be used to identify a respondent, and all project staff have signed the RTI International confidentiality agreement (Attachment E). All computers reside in a building with electronic security and are ID and password protected.

A.3. Use of Improved Information Technology and Burden Reduction

All responses will be conducted via a self-administered Internet survey. This methodology reduces respondent burden due to programming efficiencies and electronic data collection methods (i.e. point and click procedures). The use of the TRAIN system will allow us to manage our sample in an effective manner.

The survey’s data quality and control program includes skip patterns, rotations, range checks and other on-line consistency checks and procedures during the response, assuring that only relevant and applicable questions are asked of each respondent. Data collection and data entry occur simultaneously with the web-based survey. The quality of the data is also improved because the web-based survey can automatically detect errors and ensure that there is no variation in the order in which questions are asked. Data can be extracted and analyzed using existing statistical packages directly from the system, which significantly decreases the amount of time required to process, analyze, and report the data.

A.4. Efforts to Identify Duplication and Use of Similar Information

Some PHTCs have collected some basic evaluation data regarding specific courses, but no standard evaluation has been conducted of courses across all PHTCs. This survey will provide an opportunity to collect data from all PHTCs and a representative sample of PHTC courses within the HRSA TRAIN LMS.

A.5. Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this data collection.

A.6. Consequences of Collecting the Information Less Frequently

This survey evaluation is considered a one-time request for individual respondents who take courses during a specified time period (summer of 2014). However, the survey could be adapted and used over time to collect continuous evaluation data regarding the PHTC training courses.

There are consequences for not collecting these data on a frequent basis. If these data are not continually collected, PHTCs will not receive feedback regarding how the information learned during the training courses can be applied to trainee’s jobs. Additionally, PHTCs may not know how to improve course material nor will they know if additional courses are needed for training. The lack of a regular evaluation minimizes the opportunity to receive regular feedback and improve the PHTC training courses.

However, the assumption and hope is that PHTCs will continue these evaluations themselves after the data collection period ends. Aggregate data and the national level report will assist in PHTCs rating their programs vs. the national “average”. Additionally, data collected systematically across PHTCs are needed to provide a picture of the success of the PHTC program nationally.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The request fully complies with the regulation 5 CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

1. A 60-day Federal Register Notice was published in the Federal Register on January 7, 2014, vol. 79, No. 4; p. 834 (see attachment B). There was one public comment requesting further information (see attachment C).

2. In developing the data collection instrument, HRSA consulted with PHTC grantees who were engage in similar data collection efforts at individual grantee sites. The data collection instrument is based on a model provided by the PHTC evaluation workgroup, which was partially standardized among workgroup members and based on their field expertise. Individuals consulted on the appropriateness of the data collection instrument, frequency of data collection, and respondent universe include:

Melissa Sever

Ohio Public Health Training Center

[email protected]du

David Julien, Ph.D

Director of Community Planning and Evaluation

Ohio Public Health Training Center

[email protected]

Janet Place

North Carolina Public Health Training Center

[email protected]

A.9. Explanation of Any Payment or Gift to Respondents

Respondents will not receive any payment or gift.

A.10. Assurance of Confidentiality Provided to Respondents

At no time does HRSA have access to or receive potentially identifiable information. At no time is any PII linked or linkable to survey information. Only limited demographic information is requested (e.g., race, highest level of education).

The data are collected anonymously. The measures used to ensure confidentiality are described in the approved IRB protocol (Attachment D). Specifically during the consent process respondents are informed of confidentiality procedures. Additionally all members of the project staff have signed RTI International Confidentiality forms.

IRB Approval

Local IRB exemption has been obtained through the study contractor, Research Triangle International (RTI). HRSA will not have contact with study participants, nor will HRSA have access to PII. See Attachment F for a copy of the local IRB exemption letter.

10.1 Privacy Impact Assessment Information

1. Informing Respondents of the Voluntary Nature of Survey Participation:

During the written informed consent process the respondents are informed that their participation is completely voluntary and reminded that they can stop the response at any time. They are also informed that they can skip any question that they do not want to answer.

2. Opportunities to consent, if any, to sharing and submission of information:

Respondents will read a written consent on the screen before they are allowed to begin the survey. By clicking “continue” they are acknowledging they have read and understand all study procedures. The link to the web survey will only be sent via email to selected sampled respondents. The written consent will provide information on the voluntary and confidential nature of the survey, the benefits and risks of participation, the survey topic and the telephone numbers to speak with staff from HRSA or project staff from RTI International. Potential respondents are informed 1) of the purpose for the data collection; 2) that their data will be treated in a secure manner and will not be disclosed; and 3) that all information collected will be pooled with responses from other participants.

3. How the information is secured

All data are maintained in a secure manner throughout the data collection and data processing phases. Only RTI International personnel, who are conducting the study, have study-specific access to the temporary information that could potentially be used to identify a respondent (i.e., the telephone number and address). All project staff members have signed the RTI International confidentiality agreement (Attachment E). While under review, data reside on directories that only the project director can give permission to access. All computers reside in a building with electronic security and are ID and password protected.

RTI International has procedures in place to protect against data loss and down time in the event of equipment failure. These include regularly scheduled back up of data, redundant services in case of server failure, and uninterruptible power supplies to bridge a temporary loss of power. Under normal operating conditions, a complete backup of all files on every disk are written to tape weekly. Every business day, a differential backup is performed of all files created or modified since the last complete backup. In the event of a hardware or software failure, files can be restored to their status as of the time of the last differential backup, usually the evening of the previous business day. Tapes from complete backups are kept for approximately 3 months. Tapes or CD-R drives are used for long-term data archiving.

4. System of records is being created under the privacy act.

A.11. Justification for Sensitive Questions

The most sensitive questions that will be asked would be basic demographic questions about the respondents. These questions are important for statistical purposes to understand the types of people that take specific courses and to determine differences in opinion based on specific characteristics.

Other questions in the survey are more general and are satisfaction questions based on their experiences in the training course. Attachment G contains the survey instrument.

A.12. Estimates of Annualized Burden Hours and Costs

A.12.a)

HRSA is requesting 1 year of data collection for this request. The estimated number of participating respondents is 1,569. It will take approximately 10 minutes to complete the survey.

We estimate the total burden for this group to be 251 hours.

Form Name	Number of Respondents	Number of Responses per Respondent	Total Responses	Average Burden per Response (in hours)	Total Burden Hours
CE Survey Form	1,569	1	1,569	.16	251
Total	1,569	1	1,569	.16	251

A.12.b) Estimated Annualized Burden Costs

Total costs for all respondents are summarized below based on the Bureau of Labor Statistics mean hourly wage rate, reported from the May 2012 National Occupational Employment and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm#19-0000). Total burden hours are based on an estimate of 1,569 completed responses, which provides statistical power of 80% with a discernable difference of 5% between groups.

Table 3: Estimated Annualized Burden Costs to Total Respondents

Type of Respondent	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Health Educators	251	$25.53	$6,408
Total			$6,408

A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

This data collection activity does not include any other annual cost burden to respondents, nor to any record keepers.

A.14. Annualized Cost to the Government

The contract to conduct the survey was awarded to RTI International through competitive bid. The total annualized cost is $$167,612

Costs for this study include personnel for designing the study, developing, programming, and testing the survey instrument; drawing the sample; collecting and analyzing the data; and reporting the study results. The government costs include personnel costs for federal staff involved in the oversight, study design, and analysis, which include approximately 25% of a GS-13, Step 3 Public Health Analyst at a rate of $46.00 per hour.

Table 4. Estimated Annualized Cost to the Government

Type of Cost	Description of Services	Annual Cost
Government Public Health Analyst (25%)	Project oversight, study and survey design, sample selection, data analysis, and consultation	$23,750
Subtotal, Government Personnel		$23,750
Design Methodology	Contractor Labor	10,081
Survey Administration	Contractor Labor	$34,977
Report Drafting	Contractor Labor	$38,041
Fee		$21,863
Indirect Costs		$62,650
Subtotal, Contractor		$167,612
Total Annual Estimated Costs		$191,362

A.15. Explanation for Program Changes or Adjustments

This is a new information collection request.

A.16. Plans for Tabulation and Publication, and Project Time Schedule

Data analysis will be conducted to understand the influence of the PHTCs on the public health workforce and the value provided by the training courses. The questionnaire will contain general training program quality items as well as course-specific items to assess increases in topic-area knowledge among the training program participants. We will prepare tables, cross-tabulations, correlations, charts and figures, and regression models, as appropriate, as part of the analysis. Statistical testing will be used to assess the increase in topic-area knowledge after completing the course. The statistical testing will also include testing of overall satisfaction with the training program, course availability, instructors, and general value to the workforce. We will provide subgroup analyses (e.g., gender, age group) to determine whether there are differences in satisfaction and knowledge increases for different areas of the workforce. All analyses will be presented in statistical form.

Table 4. Project Time Schedule

data collection - activities	Time Schedule
Initiate email contact	Beginning immediately after OMB approval
Clean and edit data set	3 months after OMB approval
Conduct analyses	3-5 months after OMB approval
Prepare and distribute reports	7 months after OMB approval

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

The display of the OMB expiration date is not inappropriate due to the online modality of the survey instrument.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

Appendix A – Authorizing Legislation: Title VII, Sections 765 and 766 of the Public Health Service Act (42 U.S.C. 295a), as amended by the Patient Protection and Affordable Care Act of 2010, Public Law 111-148.

SEC. 766. PUBLIC HEALTH TRAINING CENTERS.

(a) In General.--The Secretary may make grants or contracts for

the operation of public health training centers.

(b) Eligible Entities.--

(1) In general.--A public health training center shall be

an accredited school of public health, or another public or

nonprofit private institution accredited for the provision of

graduate or specialized training in public health, that plans,

develops, operates, and evaluates projects that are in

furtherance of the goals established by the Secretary for the

year 2000 in the areas of preventive medicine, health promotion

and disease prevention, or improving access to and quality of

health services in medically underserved communities.

(2) Preference.--In awarding grants or contracts under

this section the Secretary shall give preference to accredited

schools of public health.

(c) Certain Requirements.--With respect to a public health

training center, an award may not be made under subsection (a) unless

the program agrees that it--

(1) will establish or strengthen field placements for

students in public or nonprofit private health agencies or

organizations;

(2) will involve faculty members and students in

collaborative projects to enhance public health services to

medically underserved communities;

(3) will specifically designate a geographic area or

medically underserved population to be served by the center that

shall be in a location removed from the main location of the

teaching facility of the school that is participating in the

program with such center; and

(4) will assess the health personnel needs of the area to

be served by the center and assist in the planning and

development of training programs to meet such needs.

Appendix B – Published 60-Day Federal Register Notice

See attachment

Appendix C – Public Comment from 60-Day Notice

See attachment

Appendix D – Institutional Review Board (IRB) Exemption

See attachment

Appendix E – RTI Confidentiality Agreement

Initial Assessment and Evaluation of Public Health Training Center Programs

CONFIDENTIALITY AGREEMENT

I, (print employee’s name), an employee of _________________________________, agree to work on the Initial Assessment and Evaluation of Public Health Training Center Programs (PHTC) in accordance with the guidelines and restrictions specified below. I understand that compliance with the terms of this agreement is a condition of my assignment with the Initial Assessment and Evaluation of Public Health Training Center Programs (PHTC) and that these terms are supplementary to those listed in my contract of employment with _________________________________.

a. I agree to treat as confidential all case-specific information obtained in the Initial Assessment and Evaluation of Public Health Training Center Programs (NISVS) and related matters. I further agree that this covenant of confidentiality shall survive the termination of this agreement.

b. I further understand that failure to follow the guidelines below may result in a potential violation of the provisions of the Privacy Act of 1974 (violation of the Privacy Act is a misdemeanor and may subject the violator to a fine of up to $5,000), and potential Institute disciplinary action, including termination. To fulfill confidentiality obligations, I will:

1. Discuss confidential project information only with authorized employees of the Initial Assessment and Evaluation of Public Health Training Center Programs (PHTC).

2. Store confidential project information as specified by project protocols.

3. Safeguard combinations, keys, and rooms that secure confidential project information.

4. Safeguard confidential project information when in actual use.

5. Immediately report any alleged potential violations of the security procedures to my immediate supervisor.

6. Not photocopy or record by any other means any confidential project information unless authorized by project leaders or my supervisor.

7. Not in any way compromise the confidentiality of project participants.

8. Not allow access to any confidential project information to any unauthorized person.

9. Report any lost or misplaced confidential project information to my supervisor immediately.

Employee’s Signature _________________________________ Date __________________

Employee’s Organization: _________________________________

Appendix F – Informed Consent Language

You have been randomly chosen to participate in an important study. The Health Services and Resources Administration (HRSA) is conducting a study of approximately 3,000 people who have completed training courses facilitated by Public Health Training Center (PHTC) programs across the country that collaborate with accredited schools of public health, academic institutions, and public health agencies and organizations. Participants will be asked questions regarding their experiences with the courses. The data we are collecting will be used to improve PHTC training courses, with the ultimate goal of improving the skills of the public health workforce.

The survey will take approximately 10 minutes to complete via the self-administered web-based questionnaire. There are minimal benefits or risks to being in this voluntary study. You can skip any question or stop the survey at any time. In order to keep your information private, the answers you give us will be combined with the answers from other people who are in the survey. No information that could personally identify you will be given to HRSA or anyone else. Your name and email address will not be connected to the information you provide in the survey, and will be erased from our files at the end of this study.

If you have questions or concerns about participating in the study, you may call Brian Evans at RTI International at 1-800-334-8571 x27366 or at [email protected]. If you have any questions about your rights as a research participant, please contact RTI’s Office of Research Protection toll-free at 1-866-214-2043.

Appendix G – Questionnaire

PHTC Participant Survey

Section 1: Demographics and Background of the course

Which of the following best describes your job?

1. Aaaaaa

2. Bbbbbb

3. Cccccc

4. Ddddd

5. Eeeeee

6. ffffffff

What is your gender?

1. Male

2. Female

What is the highest level of education that you have completed?

1. Some college

2. Associate’s degree

3. Nursing degree

4. 4-year college degree

5. Some graduate college

6. Master’s degree

7. PHD

8. Other

What institution offered this course?

How did you hear about the course? (Select all that apply.)

	Yes	No
I received an email about the course.		
I read about the course on a website.		
My manager told me about the course.		
My co-worker told me about the course.		
HRSA TRAIN search		
I heard about the course some other way.		
Please specify:

What are the top two reasons you took the course? (Select two.)

• I was required to take the course.

• My manager recommended that I take the course (but it was not required).

• I wanted to improve specific knowledge or skills.

• I was generally interested in the course content.

• Some other factor motivated me to participate in the course. Please specify: __________.

Before the course began, how useful did you think it would be for your job?

• Very useful

• Somewhat useful

• Not at all useful

• I didn’t know how useful the course would be

How was this course administered?

• Webinar

• Teleconference

• In-person

• Other

How satisfied were you with the course administration mode (e.g., in-person, webinar, teleconference)?

• Very satisfied

• Somewhat satisfied

• Neither satisfied nor dissatisfied

• Somewhat dissatisfied

• Very dissatisfied

Section 2: General Satisfaction

How knowledgeable was the instruction about the topic of the course?

• Very knowledgeable

• Somewhat knowledgeable

• Not at all knowledgeable

How prepared was the instructor?

• Very prepared

• Somewhat prepared

• Not at all prepared

How understandable was the message the was trying to convey?

• Very understandable

• Somewhat understandable

• Not at all understandable

How engaging was the instructor during the course?

• Very engaging

• Somewhat engaging

• Not at all engaging

Do you feel you were given ample opportunity to practice the skills you learned during the course?

• Yes

• No

Please indicate how much you agree with each of the following statements about the your experience in this training course.

	Strongly Agree	Agree	Neither Agree nor Disagree	Disagree	Strongly Disagree
I was satisfied with the course overall.	O	O	O	O	O
The topics discussed were important to me.	O	O	O	O	O
This course enhanced my knowledge of the subject matter.	O	O	O	O	O
This course was applicable to what I might be expected to do in my job.	O	O	O	O	O
The course provided content that it relevant to my daily job.	O	O	O	O	O
I would recommend this course to others.	O	O	O	O	O
This course met my individual needs.	O	O	O	O	O
I was provided the necessary materials to complete the course.	O	O	O	O	O
The course presentation provided ample interaction between instructor and students.	O	O	O	O	O
The materials used in the course were useful.	O	O	O	O	O
I felt comfortable asking questions during the course.	O	O	O	O	O
My questions were answered adequately during the course.	O	O	O	O	O
Overall the course met my expectations based on what I understood the objectives of the course were before the course began	O	O	O	O	O

Do you plan to/have you already recommended this course to a peer or colleague?

Yes, definitely

Maybe

No

Overall, how satisfied were you with this course?

Very satisfied

Somewhat satisfied

Neither satisfied nor dissatisfied

Somewhat dissatisfied

Very dissatisfied

Section 3: Knowledge Acquisition

To be developed by the instructors of the courses.

3-5 multiple choice questions.

Format:

1. Question question question question?

   1. Answer choice

   2. Answer choice

   3. Answer choice

   4. Answer choice

Section 4: Integration and Application

How often will you be able to use what you’ve learned in this course in your current job?

• Daily

• Once or twice per week

• Once or twice per month

• Once or twice per year

• Never

There are specific skills or knowledge that you acquired in the course that you will be able to (have been able to) use in my job. How strongly do you agree with this statement?

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

How has your work behavior changed due to the skills and/or knowledge your received in this course? Check all that apply

• answer

• answer

• answer

• answer

• answer

How will you apply what you learned in the course to your current job?

_______________________________________________________________________

How has this course helped you perform the roles and responsibilities of your current job?

_______________________________________________________________________

1 PHTC’s four statutory requirements are: (1) Establish or strengthen field placements for students in public or nonprofit private health agencies or organizations; (2) Involve faculty members and students in collaborative projects to enhance public health services to medically underserved communities; (3) Specifically designate a geographic area or medically underserved population to be served by the center that shall be in a location removed from the main location of the teaching facility of the school that is participating in the program with such center; and (4) Assess the health personnel needs of the area to be served by the center and assist in the planning and development of training programs to meet such needs. (See Title VII, Sections 765 and 766 of the Public Health Service Act (42 U.S.C. 295a), as amended by the Patient Protection and Affordable Care Act of 2010, Public Law 111-148.)

2 Kirkpatrick, D. and J. Kirkpatrick (2006). Evaluating Training Programs: The Four Levels. Third edition. San Francisco: Berrett-Koehler Publishers.