Assessment Protocol

Tanya Jennings, Public Health Advisor Science Integrity Branch (SIB), Division of Global HIV and TB (DGHT), Centers for Disease Control and Prevention (CDC), Atlanta, GA USA (SEV #8973)

Principal Investigator (PI) will be responsible for the coordination and oversight of protocol development, assessment design, project implementation, and data collection, analyses, and dissemination of results.

CDC Co-Investigators

Carin Molchan, Health Scientist Science Integrity Branch (SIB), Division of Global HIV and TB (DGHT), Centers for Disease Control and Prevention (CDC), Atlanta, GA USA

Debra Mosure, Associate Director for Epidemiological Science Scientific Integrity Branch (SIB), Division of Global HIV and TB (DGHT), Centers for Disease Control and Prevention (CDC), Atlanta, GA USA

Katina Pappas-DeLuca, Health Scientist Science Integrity Branch (SIB), Division of Global HIV and TB (DGHT), Centers for Disease Control and Prevention (CDC), Atlanta, GA USA

Letha Thomas, Clinical Research Project Manager, Science Integrity Branch (SIB), Division of Global HIV and TB (DGHT), Centers for Disease Control and Prevention (CDC), Atlanta, GA USA

Brandi Collins Thomas, Sr. Public Health Advisor, Division of Global HIV and TB (DGHT), Centers for Disease Control and Prevention (CDC), Atlanta, GA USA

Consultants

Geoff Bieger, Consultant, Deloitte Consulting, LLP, Atlanta, GA USA

Chris Palazzola, Consultant, Deloitte Consulting, LLP, Atlanta, GA USA

Sarah Morris, Consultant, Deloitte Consulting, LLP, Atlanta, GA USA

Marcel Foster, Consultant, Deloitte Consulting, LLP, Atlanta, GA USA

II. Project Overview

CDC’s Human Research Protection Office (HRPO) requires all CDC investigators to complete basic training on human research protections before they become engaged in human subjects research at CDC. This includes in-country locally employed staff and contractors. Moreover, DGHT also requires basic human research protections training for all staff who are: (1) Involved in data collection, management, and analysis activities, regardless of whether it is determined to be a research or nonresearch activity; (2) Serving as a technical reviewer or providing clearance of technical reviews for protocols, abstracts, or manuscripts; or (3) Serving as a reviewer of funding documents for the purposes of identifying potential human subjects data collection activities.

The purpose of this protocol is to describe the assessment of all Science Integrity Branch (SIB)-administered DGHT trainings provided to CDC staff and implementing partners (IPs) at headquarters and in country offices. These trainings are delivered to inform and instruct staff and partners on how to perform the activities associated with conducting and publishing research funded by DGHT. To verify and maintain a high quality of training, an assessment (see Appendix) will be provided to each participant upon completing an SIB-administered training. The data collected from the assessment will drive training improvements to best serve the in-country staff and implementing partners charged with conducting the scientific research critical to meet the goals set out by the Presidents Emergency Plan for AIDS Relief (PEPFAR) to control and ultimately end the HIV/AIDS epidemic.

III. Introduction

As the principal advisor on standards related to scientific activities within DGHT, supporting both headquarters and country programs, the DGHT SIB mission is to contribute to the broader scientific body of knowledge in global public health, provide leadership and support for PEPFAR’s Implementation Science activities, and to ensure that scientific integrity, excellence, and public health ethics are maintained in DGHT protocols, manuscripts, abstracts, and applications for funding. Overall, the main responsibilities of SIB are to provide training, capacity-building, and technical assistance for human subjects protection and scientific ethics; coordinate Implementation Science activities; review funding documents to determine whether there are potential human subjects data collection activities; coordinate scientific and ethical review of protocols; and to oversee clearance for scientific information products.

SIB routinely conducts training activities for implementing partners and DGHT staff to make sure that the workforce is adequately trained on issues related to human subjects. These trainings are conducted by SIB staff either at headquarters (Atlanta, GA) or in country and are offered to both DGHT staff as well as implementing partner staff.

IV. Justification

The purpose of this protocol is to describe the process of assessing SIB training activities. The assessments will be used to determine if the scope, duration, topics, and flow of the trainings are appropriate and to assess participant satisfaction with the training. Training modules may be modified based on the results of the assessments. Results from the assessments may also be summarized and presented in relevant conferences or technical meetings as appropriate.

V. Objective

The objective of this protocol is to assess trainings offered by SIB to determine if they provide the necessary information and tools for DGHT staff to conduct their jobs and to provide a mechanism for feedback and analysis of the trainings. By collecting feedback and data on SIB-administered trainings in a standardized manner, SIB will be able to conduct analysis to improve trainings as well as provide information to CDC leadership on the impact of SIB’s training activities.

VI. Methods

SIB has developed a variety of trainings for CDC DGHT to inform and provide instruction on human subjects research, processes, and policies to DGHT HQ staff, country office staff, and implementing partners.

DGHT staff training

The SIB training catalog offers multiple trainings in each of the following categories and includes over 30 unique modules.

Training categories include:

Protocols (Protocol Development and Approval Process, Research vs Nonresearch)
Monitoring/Project management (Good Clinical Practice, Study Monitoring)
Publications (DGHT publication/abstract submission process)
Restrictions (Review and Release of Restrictions )
Institutional Review Board (IRB Member Training)
Library Resources (CDC library trainings that are relevant for DGHT staff)
DGHT Trainings (orientation materials and CDC trainings)

These trainings are offered to all CDC DGHT staff upon joining DGHT or when a request is made from a Branch or Country office. These modules are updated on a regular basis, based on changes in policies and procedures. In-country ADSs and ADS liaisons routinely take these trainings as part of their on-boarding to DGHT. The ADS or ADS liaison training is facilitated over a two-week period in Atlanta and includes meeting with key staff members and training on a variety of topics.

Implementing partner (IP) training

“Implementing partners are the recipients of money, property, services, or anything of value to accomplish a public purpose of support or stimulation authorized by Federal statute. Substantial involvement is anticipated between the executive agency, acting for the Federal Government, and the implementing partner during performance of the contemplated activity.”¹ Thus, “Centers for Disease Control and Prevention requires all employees, contractors, and awardees conducting data activities to comply with all applicable statutes and regulations, whether the activities are for research or public health practice or occur in domestic or global settings. Moreover, for those activities that are subject to federal regulations for the protection of human research participants, data collection may not begin without the appropriate approval or clearance.”² As a result of these regulations, SIB also provides trainings to implementing partners when needed. These training courses consist of key areas:

Overview of Human Subjects Protection

Award Life Cycle
Cooperative Agreement Review
Restrictions
Protocol Development
Protocol Review/Approval
Restrictions Release
Study Implementation
Publication

These modules are periodically revised based on changes in policies and procedures or specific country office requests.

At the completion of each training (DGHT staff and implementing partner training), an assessment of the session will be initiated. The assessment form (Appendix) will be provided to each participant in order to receive participant feedback regarding the training activities conducted. Completion of the training assessment forms will not be required but will be requested of all participants.

VII. Participants

Participants will include all DGHT staff members and implementing partner staff who attend an SIB training. These trainings are offered on an on-going basis and by request from HQ or in-country leadership or from individual staff members.

VIII. Assessment Activities

In order to receive feedback from participants, a brief assessment will be administered to DGHT staff or implementing partner staff (Appendix) at the completion of each training course. Completing the assessment will be voluntary and no names of participants will be collected. The month and year of the training, the course name(s), and instructor name(s) will be collected on the form. Training participants can respond to any, none, or all of the questions. Once completed, the instructor will collect the forms and provide them to the DGHT SIB Training Unit lead. Results will be entered into the SharePoint site. The paper forms will then be destroyed.

As most IP training sessions have a large number of participants, confidentiality is not difficult to maintain. However, some SIB-administered trainings, e.g., HQ-based Country ADS training, are one-on-one; although names will not be collected on the assessment, the participant would be easily identified using other information collected. Therefore, to maintain anonymity, all assessment data will be entered into the SharePoint site, stored in a SharePoint list, and will be exported and analyzed only in the aggregate.

IX. Sample size

The sample size will vary depending on the type of training, e.g. individual versus group.

X. Data Collection

The assessment will be distributed as a paper form at the end of each training. The completed forms will be transported back to Atlanta if training is done in country. Assessment data will be entered into a SharePoint form on a CDC-HQ computer (by the study PI), and then verified for accuracy by a separate SIB staff member. Data will be stored on the SIB SharePoint site, which is only accessible by staff internal to SIB.

The training assessment process and tool will be piloted during one training session to assess the effectiveness of the assessment process as it is described in this protocol. After the completion of the pilot, results will be reviewed to determine if changes to the protocol are required before routine data collection begins.

XI. Data Management

The data will be entered into an SIB SharePoint site on a CDC-HQ computer (used by the project PI). All data will be stored within the secure SharePoint site. Data will be exported for analysis and reporting by the SIB Training Unit, and will be available for review by other SIB staff members upon request. Requests for access to data should be made to the PI.

XII. Data Analysis

Analysis will be completed using Excel or other statistical software after the data are exported from the SharePoint site. Overall frequencies will be calculated addressing the relevant topic for the assessment. Several questions may allow for free text responses. Responses will be recorded in the SharePoint site, commonalities will be assessed, and frequencies will be generated to identify patterns. Missing data will be recoded as missing and included in the denominator in most cases.

For each of the scale items, the proportion of participants who agreed and strongly agreed will be calculated. Scores (both Likert scale and open-ended questions) will be reviewed and any element receiving a score below 2 on a scale of 1 to 4 will be discussed with the trainer and appropriate changes made to the training material, as needed. Scores for individual training sessions will be maintained over time to determine if the scores are improving.

Participant answers to the free response questions will be aggregated and qualitatively analyzed to identify themes for improvement of the trainings and understand which modules are most and least valued by participants.

XIII. Data Handling and Storage

The PI and SIB will be responsible for the data and its safeguard. All efforts will be made to safeguard the data and to ensure confidentiality; the data management and security procedures ensure that the assessment information is confidential, and that only authorized personnel have access to the information.

XIV. Data Security

Responses to the surveys will be stored on the Science Integrity Branch’s SharePoint site, which can only be accessed by DGHT staff and is located behind the CDC firewall. The SharePoint list that will house the training assessment data is available to SIB staff only. This SharePoint list will include the training event name, training modules conducted during the training event, month/year of training, location (country or HQ) as well as assessment responses. Country names will be included in the SharePoint list; however, the name of the respondents will not be collected. All data will be stored on SharePoint and will only be accessible by SIB staff. Paper copies of the assessments will be shredded once the data have been entered into the SharePoint site and confirmed as entered accurately by a second SIB staff member.

Data Ownership and Sharing

The data will be owned by CDC-headquarters and results will be disseminated to CDC offices of participating branches and countries where trainings were administered upon request. All data disseminated to CDC-country office personnel will use aggregated data.

XV. Dissemination, Notification and Reporting of Results

A report will be shared with the DGHT senior staff to highlight the findings of all training assessments. These findings may also be disseminated through scientific manuscripts and conferences. Only aggregated findings will be reported. No data which can be linked to individuals will be available because no identifying information is being collected. Data may be organized and presented by country or branch when appropriate.

XVI. Challenges and Limitations

The main limitation of these training assessments will be the low number of participants for certain training modules (e.g., one-on-one HQ ADS trainings).

XVII. Ethical Considerations

All participation will be voluntary and participants will have the opportunity to opt out of completing the assessment at any point. Participants will be made aware of the voluntary nature of the assessment. Completion of the training course, including the participant certificate, will be available to each participant regardless of their completion of the assessment tool.

XVIII. Confidentiality

All paper forms will be stored in locked offices until entered into the SharePoint form and then will be destroyed after confirming that the data were entered accurately. As no personal identifiers are collected during the assessment, respondents’ identities will not be captured or reported. In the case where there may be only a single participant in a training, all data will be aggregated so identities may not be deduced by date or other information captured in the assessment. The complete data set (without respondent names) will be stored on the SIB SharePoint site behind the CDC firewall accessible to only SIB staff.

XIX. Potential Risk

There is little risk to study participants. Responses reflect the participants’ training experiences. Data collected are not of a sensitive nature.

XX. Unanticipated or Adverse Events

Any unanticipated or adverse events will be reported immediately upon notification that the event has occurred. An event could involve loss of data, for example, due to theft of the computer. As with any event, this unlikely occurrence will be reported to CGH as well as the law enforcement agencies if appropriate.

XXI. Timeline

Once all approvals of the protocol and the assessment tool have been received, the assessment tool will be piloted during the next scheduled training session. Upon completion of the pilot, results will reviewed to determine if any changes to the protocol are required. Any changes to the protocol will initiate an amendment process. After completion of the pilot, including any subsequent amendments, routine data collection may begin and will be ongoing.

XII. Appendix

Training Assessment Form Course Title(s): Training Module(s): Instructor(s): Date:
Instructions: Upon completion of the training, participants are encouraged to complete this assessment form. CDC will use this information to assess the effectiveness of training content, instructors, and methods. Please circle the response below that best describes your assessment of the training. If a question is not applicable to your training course or if you do not have sufficient information to answer, select N/A. This assessment is voluntary and no names will be identified.
SECTION I: COURSE CONTENT
	Not Applicable	Strongly Disagree	Disagree	Agree	Strongly Agree
1. The course content supported the overall learning objectives.	N/A	1	2	3	4
2. The course information was at an appropriate level to understand the learning objectives.	N/A	1	2	3	4
3. The course provided opportunities to practice and reinforce what was taught.	N/A	1	2	3	4
4. The training was relevant to the knowledge I need to accomplish my job.	N/A	1	2	3	4
5. The training increased my knowledge on the topic(s) addressed.	N/A	1	2	3	4
6. I will apply what I learned today in my work with CDC.	N/A	1	2	3	4
7. I am satisfied with this course.	N/A	1	2	3	4
8. I would recommend this course to someone (even if it were not a required course).	N/A	1	2	3	4
SECTION II: INSTRUCTOR ASSESSMENT
9. The instructor(s) was/were prepared for class.	N/A	1	2	3	4
10. The instructor(s) was/were knowledgeable about the course content.	N/A	1	2	3	4
11. The instructor(s) was/were responsive to questions and other needs.	N/A	1	2	3	4
12. The instructor(s) encouraged a participatory and interactive learning environment.	N/A	1	2	3	4
SECTION III: COURSE LOGISTICS
13. Time allotted for the overall course was appropriate.	N/A	Too Short	Adequate	Too Long	Unsure
14. Adequate time was provided for questions and discussion.	N/A	Too Short	Adequate	Too Long	Unsure
SECTION IV: ADDITIONAL COMMENTS
15. Please list the modules you found most useful and why.
16. Please list the modules you found least useful and why.
17. What suggestions do you have for improving the course?
18. Are there any additional topics you would like to see added to the course?
19. Additional comments?

1 Federal Grant and Cooperative Agreement Act of 1977: https://www.gpo.gov/fdsys/pkg/STATUTE-92/pdf/STATUTE-92-Pg3.pdf

² Reference: A Guide to Selected Scientific Regulatory Services at CDC/ATSDR, September 2006

Assessment of DGHT Science Integrity Branch Trainings, Version #1 10

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-20

Assessment Protocol

Training Assessment Protocol 20180809_clean.docx

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery