SSA Eval Technical Assistance Hub Overdose Prevention_revised_5_24_21 final

The Division of Overdose Prevention (DOP), at Centers for Disease Control and Prevention (CDC) requests a 3-year OMB approval to support the evaluation of technical assistance (TA) provided for the Overdose Data to Action (OD2A) program. OD2A is a cooperative agreement (CDC-RFA-CE19-1904) funded in 2019 to focus on comprehensive and interdisciplinary opioid overdose prevention efforts in 47 state health departments, 16 localities, Puerto Rico, Washington D.C., and the North Mariana Islands (Attachment 1). This program consists of two required components– a surveillance component and a prevention component. OD2A recipients implement a combination of activities across ten strategies within these components in order to gain access to high quality, complete, and timelier data on opioid prescribing and overdoses and to use those data to inform prevention and response efforts in their jurisdictions.

Training and technical assistance (TA) is essential to building knowledge and strengthening the capacity of recipients to implement and evaluate OD2A program strategies. CDC will develop and deploy a TA hub (hereafter referred to as the OD2A TA Center) to deliver comprehensive technical assistance and training to support the successful implementation and evaluation of surveillance and prevention activities. The OD2A TA Center is designed to enhance the efficiency, coordination, and effectiveness of TA efforts by streamlining and centralizing the provision of overdose surveillance and prevention TA. TA to OD2A recipients is divided into four different levels with multiple modes of TA delivery and involves a wide range of TA providers including CDC staff, internal and external subject matter experts (SMEs) and program partners as well as ICF staff. The four TA levels below are used to direct the process for engaging stakeholders to support program recipients and triage appropriate resources to support their needs.

TA Levels

Level 1. Individual or direct TA provided by CDC State Support Team: DOP OD2A Project Officers, Prevention Science Officers, Surveillance Science Officers. These individuals serve as the first line of TA for OD2A recipients.

Level 2. Individual or direct TA provided by DOP Strategy Leads, DOP SMEs.

Level 3. Individual or direct TA provided by an SME external to DOP, such as an SME from another CDC division, or a CDC partner and/or an ICF funded SME (Includes direct TA encounters).

Level 4. TA provided by CDC or ICF to a group of recipients. For example, webinars, site visits, peer exchanges, and virtual meetings.

The OD2A TA approach incorporates an evidence- and practice-based framework and principles to guide prioritization, design, and development. As shown in Exhibit 1, there are four core components involved in the TA framework, each building on the preceding and creating a feedback loop over time: Assess OD2A Recipient Capacity (Phase 1); Plan and Design TA and the TA Hub (Phase 2); Coordinate, Deliver, and Track TA (Phase 3); and Evaluate and Report (Phase 4). Phase 4 allows the unique opportunity to focus on monitoring, evaluation, and continuous quality improvement activities and ensure a flexible responsive TA approach to evolving needs during the OD2A program implementation cycle.

Exhibit 1: OD2A TA Center Framework

Understanding TA delivery and how it is used to support the OD2A program is a critical component of the framework, ensuring that recipient needs are met and that TA services result in meaningful change. There are two overarching purposes of evaluating OD2A TA: to gather real-time information to improve ongoing provision of TA and to demonstrate the overall outcomes of TA. Evaluation of TA provided through the OD2A Center will focus on outcomes that align with the provision of TA, including:

Alignment of TA to OD2A recipients’ needs
Increase in access to TA
Increase in use of TA
Recipient use of TA providers and reported changes in implementation (or plans for implementation)

The OD2A TA Center evaluation plan is comprised of two primary components:

Extant Data Sources

Implementation needs assessment via systematic document review
Online OD2A TA Center analytics and reports
CDC and recipient feedback via meetings and ad hoc calls

Primary Data Collection

Technical Assistance Feedback Form
Annual OD2A TA Survey

This request is for approval of the data collection instruments to be administered as part of the primary data collection component for this evaluation. This data collection will help inform Phase 4 of the OD2A TA Framework (Evaluate and Report) and is critical for completing the feedback loop that drives the overall TA approach (Exhibit 1). CDC is authorized to collect the data described in this request for evaluation and quality improvement purposes through Section 301(a) of the Public Health Service Act (Attachment 2).

Purpose and Use of Information Collection

Effective TA is built on a foundation of continuous quality improvement using data and feedback from the OD2A TA recipients. The information obtained through the primary data collection tools will allow TA providers to assess recipients’ experience and utility of information and resources gained through individual TA support, peer-to-peer sessions, and other group trainings. Additionally, data collection will inquire about additional program support needs, which will be used to inform and modify subsequent training and TA plans. This will ensure recipients have the support necessary to implement strategies that will ultimately improve opioid surveillance and prevention policies and practices within their communities.

The evaluation activities will address questions organized along a continuum beginning with TA planning and ending with TA outcomes, as demonstrated in the Exhibit 2 below.

Exhibit 2. Monitoring and Evaluation Questions

Shape2

To answer these process and improvement questions and continually improve the program, CDC developed two questionnaires. The surveys contain both process and outcome measures about TA access, utilization, delivery and outcomes. The TA Feedback Form and Annual OD2A TA Survey are included in Attachments 3-4 with screenshots of the web-based version in Attachment 5-6. Below is a description of the primary data collection instruments and the uses of the information collected.

Technical Assistance Feedback Form

The TA Feedback Form is a web-based survey that will be used to measure participant knowledge gain, intended application, and experience after each TA encounter implemented by external partners and SMEs (Level 3 and 4 TA). The TA Feedback Form will be administered following the close of each appropriate TA activity (e.g., Level 3 or Level 4 TA webinar, in-person event, virtual training, direct consultation).

The TA Feedback Form will gather OD2A recipient feedback regarding the perceived quality and effectiveness of each relevant TA activity. Survey items include a mix of close-ended questions to rate aspects of the TA provided as well as open-ended questions to gather detailed feedback about satisfaction and usefulness of the TA encounter and recommendations for improvement. In addition, the survey asks participants to indicate other TA items to address in the future. The survey also collects the role/profession and recipient jurisdiction of the respondent. Results from the TA Feedback Form will be presented in TA Feedback Reports shared with CDC following in-person and virtual trainings. The TA Feedback Reports will highlight requests for future TA support in addition to a summary of OD2A recipient feedback regarding the usefulness and quality of the TA provided. Additionally, TA Feedback Form results will inform adjustments in order to improve quality and utility in subsequent rounds of TA.

Annual OD2A TA Survey

The Annual OD2A TA Survey will include both qualitative and quantitative questions to assess satisfaction with overall TA and the extent to which TA supports informed implementation of OD2A strategies. The survey will be conducted twice (mid-point and final) with recipient leadership and with individuals from all OD2A recipient jurisdictions who participated in TA activities (levels 1-4). Since OD2A is a 3-year cooperative agreement, the survey will be conducted 18 months after the start of the program and again at the end of the 3-year period. The survey will be automated for each respondent to reflect the TA that has been received or accessed by the OD2A recipient.

The findings from these questionnaires will enable CDC and program stakeholders to improve delivery of TA and guide decision-making about future TA resource allocations. An Evaluation and Lessons learned report will be generated to maximize the application and benefit of the compiled feedback. The report will share program successes, a summary of TA activities, participants, barriers encountered, TA outcomes, and recommendations for improving TA and sustaining knowledge gains.

Use of Improved Information Technology and Burden Reduction

Data collection instruments will be administered via the Web to (1) minimize the burden to respondents and to the government, (2) maximize convenience and flexibility, (3) maximize participation and engagement, and (4) ensure the quality and utility of the information collected.

Data will be collected by web-based surveys through a link shared with respondents. Use of Web-based surveys decreases respondent burden by allowing for direct transmission of the instrument to and from survey respondents – as compared to alternative methods such as paper format. In addition, data entry and quality control mechanisms built into the Web-based system will reduce errors that might otherwise require follow-up, thus reducing burden compared to a hardcopy administration. Respondents will also be able to complete the survey at a time and location that is convenient for them. The web-surveys associated with the OD2A TA Center evaluation will recruit respondents to participate through an e-mail invitation that includes the Web-site URL to complete the survey, which will further increase the ease of responding. The Web-based questionnaires offer the following advantages for burden reduction:

Easy and secure access for respondents, decreasing the burden of reporting program activities.
Instant publication of survey results, with no printing, labeling, or postage costs, no lost paperwork, and no misprints.
Automatic sequencing of questions based on responses to previous questions, eliminating problems of inapplicable questions.
Error-checking to ensure the integrity of responses before they are submitted for review.

Efforts to Identify and Use of Similar Information

These questionnaires are not duplicated by other survey efforts or program monitoring activities. Additionally, as this program began in Fiscal Year 20, there are no existing data collected on TA provided to the CDC OD2A recipients that can be used to generate data that are similar to the information collected under this clearance. There are rare instances of studies evaluating TA provision on a program level and there are no other planned efforts to assess TA provided to the CDC OD2A recipients. This data collection does not duplicate any information currently being collected on the OD2A program such as the OD2A Annual Progress Report (APR) – OMB Control # 0920-1283, which focuses on the OD2A recipients’ progress in implementing overdose surveillance and prevention activities, or the Cross-Site Evaluation of OD2A program OMB Control # 0920-1330, which evaluates OD2A recipients’ success in implementing their activities. The purpose of this proposed data collection is to evaluate the technical assistance provided to OD2A recipients through CDC TA supports and the impact of this assistance on their implementation efforts.

Impact of Small Businesses or Other Small Entities

The majority of the data for this evaluation will be collected from state and local administrators, program staff, and partners affiliated with the OD2A recipients’ program. The data for this evaluation will be collected from individuals involved with public agencies, such as the Department of Health and Human Services, Department of Public Health, Department of Safety, and local health districts.

Consequences of Collecting the Information Less Frequently

It is important to gather real-time feedback to inform the ongoing TA provision that is required to support the OD2A recipients. The collection of information for the TA Feedback Form will be ongoing within 2 weeks of a relevant TA activity and the Annual OD2A TA Survey is scheduled to occur twice during the evaluation period (a mid-point and final survey). Less frequent data collection will hamper the evaluation impact and the ability to inform the OD2A TA strategy. The information collected by this proposed data collection will directly inform subsequent TA efforts and is critical for improving the effectiveness of the TA delivered. The TA Feedback Forms will provide near real-time data on the effectiveness of the TA that is delivered to OD2A jurisdictions and on the TA needs that exist, and will be used to continuously improve the content, format, and delivery of the technical assistance. Results from the annual OD2A TA Survey will provide a comprehensive understanding of jurisdictions’ TA needs and how well we’ve addressed those needs through technical assistance, and critical information around any gaps or inefficiencies in our TA approach. These data will allow us to maximize the reach and impact of our technical assistance efforts. Without this timely feedback on the effectiveness of the TA that is being delivered to OD2A jurisdictions, it is impossible to quickly adapt to changing needs and deliver the assistance in a way that works best for this audience. Without this information, it would not be possible to learn about the practices that promote effective TA, as well as address recipient TA needs in a short timeframe. Less frequent data collection would result in data gaps and would negatively impact our ability to understand TA needs across all OD2A jurisdictions, how to provide the most effective assistance, and how to maximize our TA resources for the purpose of improving the implementation of the OD2A program.

Special Circumstances Relating to the Guidelines of 5 CFR 1320.5(d)2

There are no special circumstances. The activities outlined in this package fully comply with all guidelines of 5 CFR 1320.5(d)2.

Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A 60-day Federal Register Notice was published in the Federal Register on December 11, 2020, Volume 85, Number 239, pp 80105 (Attachment 8) CDC received no public comments.
Efforts to Consult Outside the Agency:

The TA Feedback Form and Annual OD2A TA Survey were developed based on instruments previously approved by OMB (CDC Workplace Health Promotion Resource Center OMB#0920-1151) for other resource center evaluation effort.

Explanation of Any Payment or Gift to Respondents

No material or financial incentives will be provided to respondents for completing the questionnaires.

Protection of the Privacy and Confidentiality of Information Provided by Respondents

The Office of the Chief Information Officer at the CDC has determined that the Privacy Act does not apply to this information collection request. Some personally identifiable information (PII) will be collected including the respondents’ name, official role, organization, state, and date of interview. All information will be kept on secure, encrypted, password protected servers accessible only to specific project team members. The Privacy Impact Assessment (PIA) for this evaluation is attached (Attachment 9).

Information will be collected from stakeholders who oversee, implement, and support the program activities under OD2A, including state agency administrators, program staff, local health department staff, and evaluators. Data collection will include the TA utility, preferences, needs, and interests for the optimal delivery of TA content across a diverse group of stakeholders who are directly involved in building or sustaining opioid surveillance and prevention program activities. Although the name and work email address of the contact persons receiving the email with the survey link will be stored, this information will be stored separately from the survey responses and will be destroyed after administration.

Data collection does not involve the collection of sensitive, personal, and/or personally identifiable information. Respondents to the TA Feedback Form and Annual OD2A TA Survey will be recruited and sent an invitational email prior to participation in the information collection. CDC’s contracting company will use identifying information (name, email address) to make contact and send reminder emails to respondents. This information will be kept in a protected system.

The two OD2A TA Center evaluation surveys involve use of web-based data collection methods. The website does use cookies, and access to the web-based questionnaire will use a link to an anonymous survey. CDC’s contracting company will be the only organizations to collect, store, and maintain individual identifiable information. The electronic file linking the TA recipient and their email will be securely stored. All information will be stored in a password protected system and only accessible to evaluation staff. IT servers and data rooms have additional security. All hard drives on the server are encrypted.

An active consent process will be implemented that informs participants of the purpose of the evaluation, describes what participation entails, and addresses maintenance of privacy. All respondents who participate in the web-based surveys will be required to complete an electronic consent form prior to beginning the survey. Participation by all respondents is completely voluntary. Respondents will be assured that (1) their participation is voluntary (2) their responses will be kept privately and only seen by ICF, and (3) that there are no personal risks or benefits to them related to their participation.

Institutional Review Board (IRB) and Justification for Sensitive Questions

IRB Approval

The CDC National Center for Injury Prevention and Control’s OMB and human subject research officer has determined that IRB approval is not needed for this non-research project (Attachment 10).

Sensitive Questions

No information will be collected that are of personal or sensitive nature.

Estimates of Annualized Burden Hours and Costs

OMB approval is requested for three years for the TA Feedback form and Annual OD2A Survey. A subset of individuals from OD2A recipient locations that participate in TA activities will receive an email notification and a link with instructions for accessing the surveys. It is expected that an estimated total of 10,060 participants will be invited to complete the TA Feedback Form. The TA Feedback Form will take approximately 5 minutes to complete. While approximately 10,060 invitations may be distributed for the TA Feedback Form, based on past experience with satisfaction surveys a 40% response rate is expected. Given this anticipated response rate, the estimated number of completed responses is 4,024, which will include 2,012 unique individuals completing the TA Feedback Form twice on average.

The Annual OD2A Survey will be conducted twice over the data collection period and each survey will take 13 minutes to complete. The Annual OD2A survey will be distributed to up to 10 individuals from each recipient (660 individuals) for each administration, for a total of 1,320 completed surveys.

In conjunction with the Annual OD2A Survey, a group of email notifications, composed of one invitation email and two reminders to complete the survey, will be sent to up to 10 individuals from each recipient (660 individuals) for each survey administration, for a total of 1,320 groups of email notifications. Together, the invitation email and the two reminders will take 2 minutes to read for each individual.

Annualizing this data collection over three years results in an estimated annualized burden of 222 hours for all funded OD2A recipients. Table A.12-2 provides annualized estimates of burden for a 3-year period. The amount of time required to complete the questionnaires is based on estimates from experience with previous data collections of similar scope and length.

Table A.12-2 Estimated Annualized Burden to Respondents

Type of Respondents	Form Name	Number of Respondents	Number of Responses per Respondent	Average Burden Per Response (in hours)	Total Burden (in hours)
OD2A Recipients	TA Feedback Form (Att. 3)	671	2	5/60	112
OD2A Recipients	Annual OD2A Survey (Att. 4)	440	1	13/60	95
	Email invitation for OD2A Survey (Att. 7)	440	1	2/60	15
Total					222

Table A.12-3 provides estimates of the annualized cost to respondents for the collection of data. Cost estimates are based on average hourly rates for epidemiologists reported on the Department of Labor Statistics website for May 2018^¹. Total estimated cost to respondents is $7,035.18.

Table A.12-3 Annualized Costs to Respondents

Type of Respondents	Form Name	Number of Respondents	Number of Responses per Respondent	Avg. Burden Per Response (in hours)	Avg. Hourly Wage Rate	Total Cost
OD2A Program Staff and Partners	TA Feedback Form	671	2	5/60	$31.69	$3,549.28
OD2A Program Staff and Partners	Annual OD2A TA Survey	440	1	13/60	$31.69	$3,021.11
OD2A Program Staff and Partners	Annual OD2A TA Survey email	440	1	2/60	$31.69	$464.79
Total						$7,035.18

Estimates of Other Annual Cost Burden to Respondents or Record Keepers

No capital, start-up, or maintenance costs are involved.

Annualized Cost to Federal Government

Cost will be incurred by the government in personnel time for overseeing the project. CDC time and effort for overseeing the data collection and answering questions posed by the contractor and OD2A recipients are estimated at 5% each for two CDC employees and 2% for another CDC employee. The cost to the federal government for oversight and project management is $13,000 (Table A.14-1).

The contractor’s costs are based on estimates provided by the contractor who will carry out the data collection activities. With the expected period of performance, the annual cost to the federal government from contractor and other expenses is estimated to be $182,034 (Table A.14-1). This is the cost estimated by the contractor, ICF, and includes the estimated cost of coordination with DOP, data collection and technical assistance, analysis, and reporting.

The total annualized cost to the government, including direct costs to the federal government and contractor expenses is $195,034.

Table A.14-1. Annualized and Total Costs to the Federal Government

Expense Type	Expense Explanation	Annual Costs (dollars)
Direct Cost to the Federal Government
CDC oversight of contractor and project	CDC Supervisor labor costs	$13,000
Subtotal, Direct Costs to the Government per year		$13,000
Contractor and Other Expenses
Data collection, analysis, and reporting	Annual labor hours	$182,034
Subtotal, Contract and Other Expenses per year		$182,034
Total of all annualized expenses		$195,034

Explanation for Program Changes or Adjustments

This is a new information collection.

Plans for Tabulation and Publication and Project Time Schedule

ICF will develop TA Feedback Reports and a Final Lessons Learned Report to be used for program monitoring and quality improvement. CDC may disseminate the outcomes of the study within the federal government and outside of it through the development of case studies, scientific presentations, peer-reviewed publications, and tools and resources developed for opioid surveillance and prevention programs. Additional dissemination channels may include publications that are commonly read and of interest to TA providers and administrators who regularly manage overdose prevention programs.

The survey administration will begin as soon as OMB approval is received (anticipated in early 2021) and continue through the end of contract.

Figure A.16-1: OD2A TA Center Data Collection Timeline

Data Collection Activity	Administration Frequency
Technical Assistance Feedback Form	Ongoing after OMB approval
Annual OD2A TA Survey	Twice (mid-2021 and late 2023)

Reason(s) Display of OMB Expiration Date is Inappropriate

The display of the OMB expiration date is appropriate.

A. 18 Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the Paperwork Reduction Act.

1 Bureau of Labor Statistics. Occupational Employment and Wages, May 2018. https://www.bls.gov/oes/current/naics4_999200.htm Accessed January 2020.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Tony Richardson
File Modified	0000-00-00
File Created	2021-06-11

SSA Eval Technical Assistance Hub Overdose Prevention_revised_5_24_21 final

SSA Eval Technical Assistance Hub Overdose Prevention_revised_5_24_21 final.docx

Evaluation of the Overdose Data to Action Technical Assistance Hub

OMB: 0920-1355

Table of Contents

Section A: Justification for Information Collection

Annual OD2A TA Survey