SUPPORTING STATEMENT: PART B
OMB #
DOP Cross-Site Program Implementation Evaluation of Overdose Data to Action Program
September 18, 2020
Natasha Underwood
Centers for Disease Control and Prevention
National Center for Injury Prevention and Control
4770 Buford Highway NE MS F-64
Atlanta, GA 30341-3724
phone: 404-718-5902
fax: 770-488-8305
email: [email protected]
CONTENTS
INFORMATION COLLECTION PROCEDURES 3
B.1. Respondent Universe and Sampling Methods 3
B.2. Procedures for the Collection of Information 4
B.3. Methods to Maximize Response Rates and Deal with Nonresponse 5
B.4. Test of Procedures or Methods to be Undertaken 5
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and Analyzing Data 6
Data will be collected during key informant interviews (KII) or focus groups (FG) from a total of 346 respondents recruited from across all 66 OD2A funded jurisdictions (e.g., Program Managers (PM), Principal Investigators (PI), Surveillance Strategy Leads (SSL), and Prevention Strategy Leads (PSL)).
Primary recipients of this federal funding opportunity conduct work focused on: increasing comprehensiveness and timeliness of surveillance data; building state and local capacity for public health programs determined to be promising based on research evidence; making Prescription Drug Monitoring Programs (PDMPs) easier to use and access; and working with health systems, insurers, and communities to improve opioid prescribing. Further, they conduct work focused on linkages to care and other areas of innovation supported by evidence-based practice. They are, therefore, the most knowledgeable about the information being collected in this evaluation.
Respondents will be identified by jurisdiction PMs based on their knowledge of the specific topics covered by each KII or FG (see Exhibit 1). If any individual invited to participate is unable or unwilling to participate, they will be allowed to designate someone else who, provided they have the knowledge to be able to address the questions asked during the KII or FG, will respond in their stead. Recruitment of KII participants will cease if response saturation is reached.
Exhibit 1: Summary of OD2A information collection activities
Information Collection Method |
Respondent Universe |
Sample Size |
Methods for Selection |
KII Set 1 |
Respondents from 66 funded jurisdictions and associated delegates |
Year 1: N=50 (75% of jurisdictions) Year 2: N=50 (75% of jurisdictions*)
*Jurisdictions may be interviewed in both years for different strategies |
Jurisdictions will be sampled based on jurisdiction type to ensure representativeness within all three category types: states; counties/cities; and districts/territories. Most all counties/cities and districts/territories will be included since this is the first year they are funded under an opioid program within CDC.
|
KII Set 2 |
Respondents from 66 funded jurisdictions and associated delegates |
Year 1: N=24 Year 2: N=24
|
Criterion sampling will be used to identify jurisdictions that are implementing a unique or innovative activity for a strategy. Jurisdictions will also be stratified by program strategy. For each program strategy, jurisdictions will either be randomly selected or prioritized by stakeholders based on interest in a particular activity that could also be an “emerging or promising practice” in the field. Only three jurisdictions will be interviewed per program strategy. |
KII Set 3 |
Respondents from 66 funded jurisdictions and associated delegates |
Year 2: N=33 (50% of total jurisdictions)
|
Criterion sampling will be used to identify jurisdictions based on evaluation metrics and outcomes data. |
FG Set 1 |
Respondents from 66 funded jurisdictions and associated delegates |
Year 1:
*33 total jurisdictions represented (SSL and PSL pairs for each jurisdiction). |
Jurisdictions will be sampled based on jurisdiction type to ensure representativeness within all three category types: states; counties/cities; and districts/territories. Most all counties/cities and districts/territories will be included since this is the first year they are funded under an opioid program within CDC. Jurisdictions will also be sampled to ensure representation from each census region. |
FG Set 2 |
Respondents from 66 funded jurisdictions and associated delegates |
Year 2:
|
All jurisdictions will be invited to participate in this FG set. |
FG Set 3 |
Respondents from 66 funded jurisdictions and associated delegates |
Year 1:
|
Jurisdictions will be sampled based on whether they have activities that address a high-burden or high-risk group and by jurisdiction type to ensure representativeness within all three category types: states; counties/cities; and districts/territories. Most all counties/cities and districts/territories will be included since this is the first year they are funded under an opioid program within CDC. Jurisdictions will also be sampled to ensure representation from each census region |
Data will be collected using semi-structured KIIs and FGs (Attachment D and Attachment E). First, CDC will notify staff at funded jurisdictions that they may be asked to participate in KIIs and FGs by a formal email from the evaluation team requesting participation (Attachment G and Attachment H). Prior to engaging volunteers, the evaluation team will contact jurisdiction PMs to identify the individual most appropriate to participate based on his or her knowledge of the topic to be discussed (Attachment L and Attachment M). Emails to identified volunteers will explain the purpose and time commitment associated with participation, point out the voluntary nature of participation, and provide a list of potential dates and times (spanning multiple weeks) for participation. Volunteering participants will be asked to select at least three of the one-hour slots based on their expected availability. Based on respondents’ preferences, the evaluation team will use Outlook Calendar to schedule interviews and/or FGs. Each meeting invitation will include information regarding logistics, such as the date, time, location (virtual meeting instructions or in-person location), interview guide, and contact information. A further email sent to volunteering participants will include a request for permission to record audio of the participant during the session (Attachment F). If a KII participant does not agree to have the session audio-recorded, information will be recorded by a notetaker or the participant will be allowed to designate his or her responsibilities to a knowledgeable staff member of the same jurisdiction. If a FG participant does not agree to have the session audio-recorded, the individual will be allowed to designate his or her responsibilities to a knowledgeable staff member from the same jurisdiction.
To maximize resources, KIIs and FGs will be conducted either in person at the OD2A Annual Meeting or via WebEx, a virtual teleconference platform previously tested by evaluators (see Section 4. Test of Procedures or Methods to be Undertaken).i Both KIIs and FGs will be facilitated by at least two experienced members of the evaluation team; one will act as the primary interviewer while the other will focus on notetaking. These two team members will arrive or call into the conference line approximately 30 minutes before the scheduled information collection session to ensure all technologies are working correctly, set-up documents for notes, and prepare to conduct the session.
Once the respondent(s) arrive at the KII or FG location or join the WebEx meeting, the data collection session will begin. The interviewer will start with a brief introduction, confirm respondents’ consent to partake, take notes, and audio-record the session. The interviewer will then proceed to conduct the discussion using the appropriate semi-structured KII or FG guide. At the end of the session, the interview team will thank the respondents for their involvement and answer any questions they may have regarding next steps. Additionally, a follow up email will be sent relaying the same information and direct respondents to the appropriate point of contact for any follow up questions (Attachment K). After the respondents exit the meeting location or conference call, the evaluation team will spend time debriefing, updating interview notes, documenting key findings, and saving the audio-recording and relevant files to a secure, encrypted, password-protected external server.
Audio recordings from the interviews and FGs will be transcribed using the capabilities of a web-platform, such as Cisco WebEx. Transcripts will be validated by the evaluation team and uploaded into NVivo 12 qualitative data analysis software for data management and analysis.
Prior to analysis, a codebook will be developed and consist of deductive and inductive codes, their definitions, and inclusion and exclusion criteria for applying the codes. The research team will develop the preliminary coding structure using a deductive approach, meaning it will be grounded in the literature, including conceptual, participant, relationship, and setting codes . Deductive content analysis will be used as the primary research method to condense words into fewer content-related categories and provide knowledge, new insights, and a guide for action. Inductive content will be identified in a secondary analysis of the text to identify “emergent” codes that represent key concepts discussed by participants.
The qualitative data will then be coded and analyzed thematically to identify key themes that emerged across groups of interviews using NVivo 12 software. The team will pilot code several transcripts independently and compare coding decisions among experienced qualitative researchers. The group of coders will discuss discrepancies and build consistency accordingly. Coding will be iterative and include deductive codes (those that are established a priori from the evaluation questions’ indicators and domains) and inductive codes (those that emerge from the data). The group of coders will meet weekly during the coding process to review interpretations, resolve discrepancies, and add or collapse codes as needed. After all transcripts and documents are coded, the team will analyze the data to identify the range of opinions and topics, common themes across and within groups, and themes unique to each group. Quality assurance procedures include the training of coders, checking inter-rater reliability, and frequent debriefs on findings and coding questions.
The research team will use two different indices to assess inter-rater reliability: Cohen’s kappa and percent intercoder agreement. Inter-rater reliability will identify low-reliability on specific nodes between coders and the percentage of agreement. Cohen’s kappa was selected based on its wide acceptance across the social sciences research field as an appropriate measure of agreement between two coders. Cohen’s kappa coefficient reflects the degree of similarity between coders in assigning the same code to the same piece of text; it takes into account that agreement between coders might occur due to chance and is therefore a more conservative measure of agreement.1
A Cohen’s Kappa Coefficient value of over 0.75 can be interpreted as excellent agreement; we suggest reaching reliability of over 0.80 to confirm consistent use of codes. Once analysis is complete, all audio files will be deleted.
The KII guides were designed with a focus on streamlining questions by allowing the skipping of questions based on responses to previous questions, thereby minimizing response burden. Administration of the interview guides by phone allows rich qualitative data to be collected that would not be obtained through an online quantitative evaluation. In-person FGs will be held during OD2A Annual Meetings which funded recipients are obligated to attend.ii
Following the distribution of the invitation to participate in the information collection, respondents will have two weeks to schedule their respective KII or confirm their FG participation. Those who do not respond to the recruitment email within 1 week will receive a reminder email (Attachment I and Attachment J) urging them to respond. Those who do not respond within two weeks from the reminder email will be considered non-responders.
The estimate of burden hours is based on pilot tests of the KII guides performed by three public health professionals. The tests were conducted using WebEx to ensure platform functionality and test transcription capabilities.
Natasha Underwood, PhD, MPH
Health Scientist
Phone: 770-488-6431
Email: [email protected]
Kari Cruz, MPH
Health Scientist
Phone: 770-488-1573
Email: [email protected]
Stephen Tregear, PhD
Program Director, Booz Allen Hamilton
Phone: 215-272-0493
Email: [email protected]
Hema Desai, M. Med. Sci
Project Manager, Booz Allen Hamilton
Phone: 404-441-2580
Email: [email protected]
Jia Zhao, PhD
Interviewer, Booz Allen Hamilton
Phone: 302-690-1119
Email: [email protected]
Lindsey Bridwell, MPH, CHES
Interviewer, Booz Allen Hamilton
Phone: 404-683-0418
Email: [email protected]
Sandy-Asari Hogan, DrPH, MPH
Interviewer, Booz Allen Hamilton
Phone: 832-647-4072
Email: [email protected]
Naja Sobers, MPH
Interviewer, Booz Allen Hamilton
Phone: 774-573-1782
Email: [email protected]
Sarah Shoemaker-Hunt, PhD, PharmD
Title, Abt Associates
Phone: 612-360-7366
Email: [email protected]
Stephanie Frost, PhD
Title, Abt Associates
Phone: 404-946-6379
Email: [email protected]
i Following COV 19 guidance, at the time of the focus group, social distancing and public health safety measurement will be implemented, including considerations for virtual meetings instead of in-person.
ii Following COV 19 guidance, at the time of the focus group, social distancing and public health safety measurement will be implemented, including considerations for virtual meetings instead of in-person.
1 McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276-282.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Moreland, Briana (CDC/ONDIEH/NCIPC) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |