PCNASP OMB Statement B - Revised_CDC comments - 1.5.2018

Paul Coverdell National Acute Stroke Program (2015-2020) Assessment

Information Collection Request

Supporting Statement

Part B—Statistical Methods

January 5, 2018

Kincaid Lowe, Health Scientist

CDC/NCCDPHP/Division for Heart Disease and Stroke Prevention

4770 Buford Highway NE, MS-F73

Atlanta, GA 30341-3717

Phone: 404.718.6633

Email: [email protected]

Fax: 770.488.8151

Form 83-I

TABLE OF CONTENTS

B. STATISTICAL METHODS 4

B.1 Respondent Universe and Sampling Methods 4

B.2 Procedures for the Collection of Information 6

B.3 Methods to Maximize Response Rates and Deal with Nonresponse………………….8

B.4 Test of Procedures or Methods to be Undertaken 9

B.5 Individuals Consulted on Statistical Aspects and Individuals and/or Analyzing Data 10

LIST OF ATTACHMENTS – Section B

Attachment A2 Screenshots of Partner Cost Collection Tool

Attachment A3 Partner Cost Collection Tool User Manual

Attachment A5 Principal Investigator Telephonic Interview Protocol

Attachment A6 Program Manager Telephonic Interview Protocol

Attachment A7 Support Staff Telephonic Interview Protocol

Attachment A8 QI Specialist Telephonic Interview Protocol

Attachment A9 Data Analyst Interview Protocol

Attachment B1 Introductory Email to Grantees to Request Partner Contact Information (from CDC)

Attachment B2 Introductory Email and Reminders to Partners about Cost Study (from RTI)

Attachment B3 Thank You E-Mail for Partner Cost Data Collection

Attachment B4 Introductory Email to Schedule Interview Preparatory Call

B. STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

CDC and RTI International propose to collect information from all nine funded Paul Coverdell National Acute Stroke Program (PCNASP) grantees and a sample of their partners to (1) assess the actual partner costs of establishing statewide comprehensive stroke systems of care; and (2) increase CDC’s understanding of funded programs’ implementation of effective state-based stroke systems of care and PCNASP specific contributions. Two components of the information collection include: (1) program implementation cost data collection from program partners using a cost collection tool; and (2) telephone interviews with key program stakeholders. For each component, we employ purposive sampling to select participants. Each component is discussed in detail below.

Cost Collection Tool

CDC will ask each PCNASP grantee to identify a sample of their partners to invite to participate in the PCNASP partner cost data collection. To do so, CDC will email the program manager from each funded program to ask them to complete a simple table that contains contact information for a sample of partners from small, medium, and large organizations. In the table, the program manager will fill out the partner organization’s name, name of the contact person at the organization, and email address of the contact person (Attachment B1).

Two cycles of information collection will be conducted over a 3-year period. The first cycle is scheduled for April-May 2018 and the second cycle is scheduled for April-May 2019. Based on our purposive sampling approach, the annualized number of responses is 205 and the total number of responses is 410 summarized in Table B1.

Telephonic Interviews

We will recruit program staff in funded states to complete 1 hour telephone interviews designed to explore programs’ implementation of effective state-based stroke systems of care and their specific contributions (Attachment A5-A9).

Using purposive sampling, CDC will recruit the program manager from each funded program to participate in the interviews. In addition, the program manager will be asked to participate in a planning call to identify five other key staff and partners to participate in the interviews (Attachment B4). Relevant staff will include principal investigators, program managers, quality improvement specialists, data analysts/ program evaluators, and partner support staff such as post-hospital staff participating in PCNASP, heads of state EMS, or EMS responders. To obtain a variety of perspectives on program implementation issues, information will be collected from individuals in a variety of roles. A total of approximately 6 program and partner staff per site will participate in the interviews.

One cycle of information collection will be conducted over a 3-year period, in February 2019. The annualized number of responses is 54, as summarized in Table B1.

Table B1: Number of Respondents per Cycle of Data Collection, by Type of Respondent; Total Number of Responses Over the 3-Year OMB Approval Period; and Annualized Number of Responses

Type of Respondents	Number of Respondents in 2018	Number of Respondents in 2019	Total Number of Responses Over 3 Years	Annualized Number of Responses
CCT
Partner Program Manager	205	205	410	205
Key Informant Interviews
Principal Investigator	0	9	9	9
Grantee Program Manager	0	9	9	9
Quality Improvement Specialist	0	9	9	9
Data Analyst/ Program Evaluator	0	9	9	9
Partner Support Staff	0	18	18	18
TOTAL	205	259	464	259

B.2 Procedures for the Collection of Information

Cost Collection Tool

We are developing a Web-based Partner Cost Collection Tool (CCT) to collect information from program partners (Attachments A2) twice in three years. A sample of program partners from all nine grantees will be asked to participate in the cost data collection and will receive comprehensive training on using CCT. A detailed User’s Manual will be provided with the cost instrument to assist the awardees in providing the requested data accurately (Attachment A3). Training on the use of CCT will be provided via a Webinar conducted by RTI before the first data collection.

Send Initial Contact and Reminder Emails

Grantees will provide initial information about the study to partners via email (Attachment B1). RTI will invite program partners to participate in the cost study via email (Attachment B2). RTI will send one reminder email to program partners one week before data collection ends (Attachment B2). Program partners will receive a thank you email for their participation within one week of submitting data through CCT (Attachment B3). A goal will be to achieve an 50% response rate in both years 2018 and 2019.

Collect Data

Program partners will submit data using the Web-based Partner Cost Collection Tool (CCT). Automated data checks will be incorporated in the tool, and this will allow the respondents to review and check data prior to transmission.

Once program partners submit data, RTI will log and archive the data. RTI will review the data for accuracy and completeness. RTI will perform thorough data validation to assess the quality of the data available to perform the planned analysis. All data collected in CCT will be assessed for missing information (percentage of fields with missing data) and incorrect data (percentage of data elements with formats that are not recognized; percentage with inappropriate range of values). Throughout the funding period, we will also review whether the subcategories sum up to the expected total costs. Discrepancies between the total amount of funds expended for all interventions and the total itemized costs will be identified and clarified with the program partners.

Based on each program partner’s submission, a report will be produced that contains counts and associated percentages for blank field errors, inter-field relationship errors, and inter-record relationship errors, in each data set. RTI will follow-up with partners via email with any questions about their submitted data. We will then create an aggregated analysis file for generating reports and publications.

Telephonic Interviews

Information will be collected once from key program and partner staff. Information will be collected by conducting 1-hour telephone interviews of approximately 6 key informants for each grantee. Respondents will include principal investigator (1), program manager (1), quality improvement specialist (1), data analyst/ program evaluator (1), and partner support staff (2). Five versions of the Interview Guide have been developed so that questions are targeted to specific respondent groups (see Attachments A5-A9). Qualitative data from these interviews will provide the opportunity to unearth valuable details not otherwise obtainable through quantitative studies, to expand on key strategies that contributed to improved stroke systems of care and greater infrastructure to support data linkages, data collection, and data-driven quality improvement activities.

Send Initial Contact and Reminder Emails

RTI staff will work individually with each key staff member to find convenient times for them to complete the interview. Once mutual availability has been established, RTI staff will send each interviewee an electronic meeting request containing information for a secure conference line.

Collect Data

Telephonic interviews will be conducted by teams of 2 contractor staff, including at least one senior staff member and one supporting staff. All contractor staff will be well trained and experienced in qualitative methodology, including in-depth interviewing.

Respondents will be asked to grant permission for the interview team to audio record the interview for note taking and clarification purposes only. The audio tapes will be destroyed once they have been used to fill in any gaps in the notes taken by the note taker.

Prior to conducting interviews at each site, RTI will use existing data resources, such as the PCNASP grantees; funding application and progress reports, to prepare the interviewer(s) to conduct the interviews in a focused and efficient manner.

Exhibit B2. Flowchart of Data Collection Activities

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

Upon OMB approval, RTI will implement the CCT and key informant interview protocol. RTI will take the steps below to maximize responses.

Cost Collection Tool

• Identification of Key Personnel. By working with grantee program managers to identify key partners, CDC and RTI ensure they are reaching out to individuals who are appropriately involved and invested in the program.

• Advance Notice to PCNASP Grantees and Partners. CDC and RTI will notify grantee and partner point of contacts of the data collection via email about one month in advance of the anticipated start time to allow for respondents to plan their availability accordingly (Attachment B1 & B2). CDC and RTI will also request that grantees reach out directly to partners in advance to give partners even greater advance notice (Attachment B1).

• Reminder Emails. RTI will send one follow up email to program partners one week before CCT data collection ends to ensure maximum response (Attachment B2).

• Hands-on Training and Technical Assistance. Participation in the cost data collection is voluntary for program partners. To increase ease of participation, training in the use of the data reporting system will be provided to partner staff assisting with cost data collection and reporting. Program partners will also receive a User’s Manual that provides complete written instructions regarding the cost data submission requirements (Attachment A3). This document will support consistent submissions across program partners.

Telephonic Interviews

• Identification of Key Personnel. By working with grantee program managers to identify key partners, CDC and RTI ensure they are reaching out to individuals who are appropriately involved and invested in the program.

• Advance Notice to PCNASP Grantees and Partners. CDC and RTI will schedule a planning call with program managers about one month in advance of the anticipated start time to provide advance notice of the logistics of the interviews and to identify potential interviewees.

B.4 Test of Procedures or Methods to be Undertaken

Cost Collection Tool

RTI will conduct a pilot test of the CCT with 1 grantee to assess respondents’ ability to understand the data elements requested, identify the cost information required, complete the tool within the allocated timeframe, and finalize the time burden estimates.

The information learned from pretesting will be used to finalize CCT and User’s Manual. Feedback from pretesting will be incorporated to create the final Web-based cost instrument that will serve as the data collection instrument for all the program partners.

A similar Excel-based tool was used in the assessment of the 2012-2015Paul Coverdell National Acute Stroke Registry. Improvements were made to the tool and data collection protocol based on lessons learned from this assessment..

The CCT requests expenditure details for PCNASP activities.

Using the information collected through CCT, intervention cost estimates will be generated yearly, resulting in cost-effectiveness and health and cost benefit analyses that will then be integrated into the system dynamics model. Each year, the following costs will be estimated for each respondent:

• total spending in during reporting period,

• cumulative spending to date,

• cumulative spending by venue (e.g., pre-hospital, in-hospital, post-hospital)

For the final analyses, we will examine mean costs per partner and potential explanatory factors for variation in partner costs.

Telephonic Interviews

RTI will conduct pilot interviews with one grantee (six respondents) to assess respondents’ ability to understand questions on the data protocol, ability to provide information collection, and feasibility of completing the interview within a 60-minute timeframe.

Once the data are collected, RTI will conduct qualitative analysis of the interview data using QSR NVivo 11.0, the same software used in the previous assessment of the Coverdell program.

We will formulate a hierarchically ordered, analytical coding structure that reflects the key constructs (e.g., planning activities, implementation strategies, leaders’ roles and responsibilities, required resources, partners). Hierarchically arranged codes logically structure the data around research questions and concepts of interest. This, in turn, supports analysis by ensuring that qualitative data address assessment questions (e.g., link to the conceptual model) and facilitate data retrieval for concepts of interest. Each member of the qualitative analysis team will review the key constructs and codes to ensure that all coders are in agreement on what each code means.

To ensure reliable coding, we will hold a coding training, conduct a pilot coding exercise, double-code up to 20% of interviews to ensure high inter-rater reliability, and meet regularly to debrief on our coding tasks. Through the double coding process, we will aim to achieve a Cohen’s kappa coefficient of 0.75 as a measure of good reliability.

After the data are coded, RTI will link to the secondary data and analyze the results in terms of the key constructs. After completion of coding of the data at the site level, we will begin the process of examining cross-site findings for reporting and dissemination.

B.5 Individuals Consulted on Statistical Aspects and Individuals and/or Analyzing Data

CDC will provide overall direction for the PCNASP assessment data collection activities, directing regular planning and coordination meetings with RTI International staff, including the data collection protocol and data reporting.

The Partner Cost Collection Tool and telephonic interview protocols were designed in collaboration with economists and evaluators at CDC. Under contract with CDC, RTI International will recruit and administer the data collection protocol with PCNASP funded grantees and partners. RTI International will also analyze and report the CCT and qualitative results.

Other personnel involved in design of the data collection plan and instruments are listed in Table B.5-A.

Table B.5-A. Staff within the Agency and Experts Outside of the Agency Consulting on Study Design and Instruments

CDC Staff
Joanna Elmi	Phone: 770-488-5979 E-mail: [email protected]
Kincaid Lowe	Phone: 404-718-6633 E-mail: [email protected]
Ashley Marshall	Phone: 404-639-7202 E-mail: isg6@cdc.gov
John Chapel	Phone: 770-488-6538 Email: [email protected]
Jennifer Foltz	Phone: 770-488-6084 E-mail: [email protected]
Non-CDC Staff
Stephanie Teixeira-Poit	Phone: 919-541-5915 E-mail: [email protected]
Benjamin Yarnoff	Phone: 919-541-6640 E-mail: [email protected]
Janice Tzeng	Phone: 919-541-6410 E-mail: [email protected]
Devon Wachtmeister	Phone: 919-541-6149 E-mail: [email protected]

Kincaid Lowe, MPH

Health Scientist

Applied Research and Evaluation Branch

Division for Heart Disease and Stroke Prevention

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Hwy NE, MS-F73

Atlanta, GA 30341-3717

OFFICE: 404.718.6633

FAX: 770-488-8151

Information will be collected and analyzed by CDC’s contractor, RTI International.