Supporting Statement B

Supporting Statement B.docx

[NCCHPHP] Comprehensive Evaluations of theDP-23-003, DP-23-004, and DP-23-0005 Cooperative Agreement Programs: The National Cardiovascular Health Program, The Innovative Cardiovascular Health Program

OMB: 0920-1453

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0920-1453 can be found here:

2025-05-01 - No material or nonsubstantive change to a currently approved collection

Document [docx]

Download: docx | pdf

Comprehensive Evaluations of the DP-23-0003, DP-23-0004, and DP-23-0005 Cooperative Agreement Programs: The National Cardiovascular Health Program, The Innovative Cardiovascular Health Program, and The Well-Integrated Screening and Evaluation of Women Across the Nation Program (WISEWOMAN) Program

[OMB No. if applicable. 0920-xxxx] [OMB expiration date]

Supporting Statement B

Program Official/Contact

Julia Jordan

Health Scientist

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

P: (770) 488-1053

F: (770) 488-8151

[email protected]

8/12/2024

TABLE OF CONTENTS

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 4

B1. Respondent Universe and Sampling Methods 4

B2. Procedures for the Collection of Information 12

B3. Methods to Maximize Response Rates and Deal with No Response 17

B4. Tests of Procedures or Methods to be Undertaken 18

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 19

ATTACHMENTS

1. Centers for Disease Control and Prevention National Center for Chronic Disease Prevention and Health Promotion Notice of Funding Opportunity

1a. The National Cardiovascular Health Program CDC-RFA-DP-23-0004

1b. The Innovative Cardiovascular Health Program CDC-RFA-DP-23-0005

1c. WISEWOMAN: Well-Integrated Screening and Evaluation of WOMen Across the Nation CDC-RFA-DP-23-0003

2. Authorizing Legislation

2a. Public Health Service Act [42 U.S.C. 247b]

2b. Patient Protection and Affordable Care Act [42 U.S.C 300u-11]

2c. Public Health Service Act [42 U.S.C. 241a] 93.426

2d. Public Health Service Act [42 U.S.C. 300n-4a]

2e. Public Health Service Act [42 U.S.C. 300l]

2f. FY 2023 Consolidated Appropriations Act (Pub. L. 117-328, Div. H).

2g. Public Health Service Act [42 USC 300k]

3. Evaluability Assessment Data Collection Instruments

3a. Evaluability Assessment Nomination Form - National/Innovative

3b. Evaluability Assessment Nomination Form - WISEWOMAN

3c. Evaluability Assessment Interview Guide - National/Innovative Learning Collaborative

3d. Evaluability Assessment Interview Guide - CQM National/Innovative Recipient-Level

3e. Evaluability Assessment Interview Guide - CQM WISEWOMAN Recipient-Level

3f. Evaluability Assessment Interview Guide - CQM National/Innovative Partner-Level

3g. Evaluability Assessment Interview Guide - CQM WISEWOMAN Partner-Level

3h. Evaluability Assessment Interview Guide - TBC National/Innovative Recipient-Level

3i. Evaluability Assessment Interview Guide - TBC WISEWOMAN Recipient-Level

3j. Evaluability Assessment Interview Guide - TBC National/Innovative Partner-Level

3k. Evaluability Assessment Interview Guide - TBC WISEWOMAN Partner-Level

3l. Evaluability Assessment Interview Guide - CCL National/Innovative Recipient-Level

3m. Evaluability Assessment Interview Guide - CCL WISEWOMAN Recipient-Level

3n. Evaluability Assessment Interview Guide - CCL National/Innovative Partner-Level

3o. Evaluability Assessment Interview Guide - CCL WISEWOMAN Partner-Level

4. Exploratory Assessment Data Collection Instruments

4a. Exploratory Assessment Interview Guide - National/Innovative Learning Collaborative

4b. Exploratory Assessment Interview Guide - CQM National/Innovative Recipient-Level

4c. Exploratory Assessment Interview Guide - CQM WISEWOMAN Recipient-Level

4d. Exploratory Assessment Interview Guide - CQM National/Innovative Partner-Level

4e. Exploratory Assessment Interview Guide - CQM WISEWOMAN Partner-Level

4f. Exploratory Assessment Interview Guide - TBC National/Innovative Recipient-Level

4g. Exploratory Assessment Interview Guide - TBC WISEWOMAN Recipient-Level

4h. Exploratory Assessment Interview Guide - TBC National/Innovative Partner-Level

4i. Exploratory Assessment Interview Guide - TBC WISEWOMAN Partner-Level

4j. Exploratory Assessment Interview Guide - CCL National/Innovative Recipient-Level

4k. Exploratory Assessment Interview Guide - CCL WISEWOMAN Recipient-Level

4l. Exploratory Assessment Interview Guide - CCL National/Innovative Partner-Level

4m. Exploratory Assessment Interview Guide - CCL WISEWOMAN Partner-Level

5. Cost Study Data Collection Instruments

5a. Resource Use and Cost Inventory Tool - Recipient-Level

5b. Resource Use and Cost Inventory Tool - Partner-Level

5c. Cost Study Interview Guide - Recipient-Level

5d. Cost Study Interview Guide - Partner-Level

6. 60-Day Federal Register Notice

7. Non-Research Determination

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

The Centers for Disease Control and Prevention (CDC) Division for Heart Disease and Stroke Prevention (DHDSP) proposes to collect information from a recipients of three cooperative agreements and their partners:

CDC-RFA-DP-23-0004: The National Cardiovascular Health Program (The National CVH Program) (Attachment 1a)
CDC-RFA-DP-23-0005: The Innovative Cardiovascular Health Program (The Innovative CVH Program) (Attachment 1b)
CDC-RFA-DP-23-0003: Well-Integrated Screening and Evaluation of Women Across the Nation (WISEWOMAN) (Attachment 1c)

CDC has contracted with Deloitte Consulting to design and implement the Comprehensive Evaluation. Deloitte Consulting, together with the Division for Heart Disease and Stroke Prevention (DHDSP) be responsible for data collection and analysis activities. Deloitte and DHDSP are referred to collectively as the Comprehensive Evaluation Team.

The comprehensive evaluation consists of three components 1) Evaluability Assessment; 2) Exploratory Assessment, and 3) Cost Study. Participation in any component is optional; however, findings from the evaluability assessments will be used to invite participation in the exploratory assessments. Therefore, we anticipate that sub-set of recipients and their partners that participate in the Evaluability Assessment will also participate in the Exploratory Assessment. In this comprehensive evaluation, “recipient” indicates the organization funded directly by the cooperative agreement. A “recipient site” refers to the funded recipient organization, its implementation partners, and Learning Collaboratives.

Table B.1 summarizes the potential respondent universe, targeted respondents, and methods for selecting respondents for the proposed new data collection activities. Below, we discuss the targeted respondents and methods in more detail for the evaluation activities for each evaluation component.

Table B.1. Summary of the Comprehensive Evaluation Activities

Evaluation Component	Data Collection Method	Respondent Universe	Targeted Respondents	Methods for Selection
Evaluability Assessment	Nomination Form	All 107 National CVH Program, Innovative CVH Program and WISEWOMAN Recipients	Program Managers	Not applicable, all recipients will be invited to submit a nomination form
	Semi-Structured Recipient Interviews	24 representatives from The National CVH Program, The Innovative CVH Program, and WISEWOMAN recipient organizations	Program Managers, Evaluators, Data Preparers, Health Equity Specialists	Recipient sites will be selected based upon a review of nomination forms and a prioritization exercise based on a range of selection criteria
	Semi-Structured Partner Interviews	Representatives from organizations that have partnered with the National CVH Program, Innovative CVH Program, and WISEWOMAN recipients selected to participate in the Evaluability Assessment	Program Manager, Health Equity Specialist, Clinical and Field Staff (e.g., CHWs, HIT Coordinator, Team Based Care member, Patient Navigator, Quality Improvement Specialist, HBSS staff, Health Coaches)	Partner sites will be selected based on implementation capacity, partnership history, willingness to participate, and recommendations from recipients selected to participate in the Evaluability Assessment
	Semi-Structured Learning Collaborative (LC) Interviews	Representatives from the LC’s created or managed by The National CVH Program, and The Innovative CVH Program recipients selected to participate in the Evaluability Assessment	LC Lead, LC Coach, or other identified LC participant	Representatives from LCs will be based on capacity, willingness to participate, and recommendations from recipients selected to participate in the Evaluability Assessment
Exploratory Assessment	Semi-Structured Recipient Interviews	12 representatives from The National CVH Program, Innovative CVH Program, and WISEWOMAN recipient organizations that participated in the Evaluability Assessment	Program Managers, Evaluators, Data Preparers, Health Equity Specialists	Recipient sites will be selected based upon a review of Evaluability Assessment findings and a prioritization exercise based on a range of selection criteria
	Semi-Structured Partner Interview	Representatives from organizations that have partnered with The National CVH Program, The Innovative CVH Program, and WISEWOMAN recipients selected to participate in the Exploratory Assessment	Program Manager, Health Equity Specialist, Clinical and Field Staff (e.g., CHWs, HIT Coordinator, Team Based Care member, Patient Navigator, Quality Improvement Specialist, HBSS staff, Health Coaches)	Partner sites will be selected based on capacity, willingness to participate, and recommendations from recipients selected to participate in the Exploratory Assessment
	Semi-Structured LC Interviews	Representatives from the LC’s created or managed by The National CVH Program, and The Innovative CVH Program recipients selected to participate in the Exploratory Assessment	LC Lead, LC Coach, or other identified LC participant	Representatives from LCs will be based on capacity, willingness to participate, and recommendations from recipients selected to participate in the Exploratory Assessment
Cost Study	Semi-Structured Recipient Interviews	Representatives from up to 55 National CVH Program, Innovative CVH Program, and WISEWOMAN recipient organizations that volunteer to participate in the Cost Study	Cost study liaison (PI, Program Manager or other identified staff with comprehensive understanding of budget/staffing structures)	Recipients will be invited to participate via a Call for Interest process. If more than the maximum sample size responds to the Call for Interest, recipients sites will be selected using a prioritization exercise based on a range of selection criteria
	Recipient Cost Inventory Tool
	Semi-Structured Partner Interview	Representatives from organizations that have partnered with The National CVH Program, The Innovative CVH Program, and WISEWOMAN recipients that volunteer to participate in the cost study.	Program Manager, Program Coordinator, or other identified staff with an understanding of implementation activities and budget/staffing structures	Partners will be invited to participate. If more than the maximum sample size responds, then partner sites will be selected based on capacity, willingness to participate, and recommendations from recipients participating in the Cost Study
	Partner Cost Inventory Tool

Evaluability Assessments

Recipients will self-nominate to participate in the Evaluability Assessment using a Nomination Form (Attachment 3a and 3b). We anticipate a 50% response rate to the nomination form. The Comprehensive Evaluation Team will select a total of 24 self-nominated recipients using a purposive sampling approach that applies the following prioritization criteria: project officer and technical assistance (TA) evaluation provider recommendations; availability and willingness of recipients and partner sites to participate in the Evaluability Assessment and Exploratory Assessment. The selected sample will aim to have a mix of geography (rural/urban/suburban); location (city/county/state); partner type; populations of focus; implementation settings; and length of CDC funding. Each selected recipient will self-nominate to participate in the evaluation for one of three cooperative agreement strategies, 1) Track and Monitor Clinical Measures (CQM), 2) Implement Team-Based Care (TBC), 3) Link Community Resources and Clinical Services (CCL). . While the aim is to have a sample size that has an even distribution of program strategies and is proportional to the number of awards by cooperative agreement the sample size per strategy and per cooperative agreement may be revised to be based on the number of nominations received and the number of recipients who implement more than one cooperative agreement. Due to the self-nomination process, we anticipate a nearly 100% response rate for the semi-structured interviews described below. To increase the response rate, the Comprehensive Evaluation team will work with participants to schedule the interview at a convenient day and time.

Semi-Structured Recipient Interviews (Attachments 3d, 3e, 3h, 3i, 3l, 3m). For each participating recipient, key informant interviews will be conducted with up to four staff members resulting in a total maximum sample size of 78 individuals across all 24 selected recipients. Interview participants may include program managers, evaluators, data preparers, and health equity specialists. The Comprehensive Evaluation Team will work with recipient program managers/team leads to identify and connect with program staff who work closely on managing or implementing the strategies to participate in the interviews.

Semi-Structured Partner Interviews (Attachments 3f, 3g, 3j, 3k, 3n, 3o): The Comprehensive Evaluation Team will work with program managers/team leads at recipient organizations selected to participate in the Evaluability Assessment to identify partner organizations that are supporting program strategy implementation. The Comprehensive Evaluation Team will use a purposive sampling process to identify up to two partner sites for each cooperative agreement recipient participating in the Evaluability Assessment. Site selection criteria will include implementation capacity, history of collaboration with recipient organization, implementation setting, type of organization (e.g., large hospital, rural health center, etc.), and willingness of the site to participate. Key informant interviews will be conducted with up to four respondents per selected recipient site for a total sample size of 78 individuals. Partner site interviewees may include program managers, health equity specialists, and clinical and field staff (e.g., community health workers (CHWs), health information technology (HIT) coordinators, team-based care members, patient navigators, quality improvement specialists, healthy behavior support services (HBSS) staff, and health coaches).

Semi-Structured LC Interviews (Attachment 3c): The Comprehensive Evaluation Team will work with The National CVH Program and The Innovative CVH Program recipients to identify Learning Collaborative (LC) members (e.g., LC lead, LC convener, LC coach, LC participant) from their respective cooperative agreements to participate in the interview. Key informant interviews will be conducted with two respondents from each participating National CVH Program and Innovative CVH Program site for a total sample size of 36 individuals.

Document Reviews: The Comprehensive Evaluation Team will aggregate and review relevant program documents of recipients and partners selected to participate in the key informant interviews. The types of documents for review will vary by site but may include organization website, training curricula, promotional materials, and process manuals and progress and evaluation reports submitted to CDC. Document reviews will occur before interviews to help become more familiar with the sites and inform the development of logic model and probes for semi-structured interview guides.

Exploratory Assessments

Semi-Structured Recipient Interviews (Attachment 4b, 4c, 4f, 4g, 4j, 4k): The sample selection for the Exploratory Assessment will be informed by the Evaluability Assessment findings, such that the Evaluability Assessment will identify recipients from each cooperative agreement with the most promising approaches and capacity to participate in the Exploratory Assessment. A total of 12 recipients will be selected using a purposive sampling approach that applies the following prioritization criteria: time in operation/program maturity/implementation progress; populations of interest; focus on health equity; innovative approaches; evidence of effectiveness and sustainability; data system capacity (status of data availability); strength of community partners; organizational capacity; geography (rural/urban/suburban); location (city/county/state); partner type; implementation setting; and length of funding. Selected recipients will be evaluated on the same strategy for which they participated in the Evaluability Assessment (CQM, TBC, or CCL). The number of selected recipients per cooperative agreement will be proportional to the number of awards resulting in six recipients for The National CVH Program (two per strategy), three recipients for The Innovative CVH Program (one per strategy), and 3 recipients for WISEWOMAN (one per strategy).

For each participating recipient, key informant interviews will be conducted with 3-4 staff members, resulting in a sample size of 39 individuals. Interview participants will serve in similar roles as respondents in the Evaluability Assessment and may or may not be the same individual. The Comprehensive Evaluation Team will work with recipient program managers/team leads to follow up with previously participating program staff or identifying new individuals who work closely on managing or implementing the strategies to participate in the interviews.

Semi-Structured Partner Interviews (Attachment 4d, 4e, 4h, 4i, 4l, 4m):The same partner sites engaged in the Evaluability Assessment are expected to participate in the Exploratory Assessment for recipient sites selected for the Exploratory Assessment. However, if circumstances prevent the original partner sites from participation, then the Comprehensive Evaluation Team will work with the recipients to identify new partner organizations that are supporting the implementation of The National CVH Program, The Innovative CVH Program, or WISEWOMAN strategies as needed. In the case where new partner organizations are identified, the Comprehensive Evaluation Team will use a purposive sampling process so that each recipient site has up to two partner sites engaged in the Exploratory Assessment. Site selection criteria will be the same as used for the Evaluability Assessment (e.g., implementation capacity, collaboration history, willingness to participate, etc.). Key informant interviews will be conducted with up to four respondents per recipient site for a maximum total sample size of 39 individuals.

Semi-Structured Learning Collaborative Interviews (Attachment 4a): A total of nine LCs from The National CVH Program and The Innovative CVH Program will be invited to participate in an interview. Up to two staff members from each LC will be invited to participate in the interview resulting in a maximum total sample size of 18 individuals. Using the same process as for the Evaluability Assessments, the Comprehensive Evaluation Team will work with The National CVH Program and The Innovative CVH Program recipients to identify Learning Collaborative members (e.g., LC lead, LC convener, LC coach, LC participant) to participate in the interview.

Document Reviews: The Comprehensive Evaluation Team will aggregate and review new program documents developed after the Evaluability Assessment of recipients and partners selected to participate in the interviews. Document reviews will occur before interviews to help with the development of probes for semi-structured interview guides.

Cost Study

The Comprehensive Evaluation Team will invite recipients from each cooperative agreement to participate in the Cost Study though a Call for Interest. Recipients are the primary unit of analysis for the Cost Study and given that recipients will be working closely with partners to implement strategies, the secondary unit of analysis will be at the partner level where activities are implemented. Sites will be selected using a two-stage sampling approach to select a subset of recipients (primary unit of analysis) and their partners (secondary unit of analysis). Recipients will indicate their interest in participating in the Cost Study via email and do not have to participate in the Evaluability Assessment or Exploratory Assessment to participate. We aim to have a maximum of 50% recipients (n =55) participating in the cost study to allow for a robust cost analysis or a minimum of 20% (n=21) to achieve representation. However, we anticipate that 40% of recipients will volunteer to participate in this data collection, as this is a newer type of data collection for these recipients. While we aim to have 55 recipient sites participating, collecting data from a smaller number of recipient sites will still provide CDC with new and key insights into the cost of program implementation.

Should more than 55 recipients indicate interest in participating in the Cost Study, we will apply prioritization criteria to select the final sample. Prioritization criteria includes the following: project officer and TA evaluation provider recommendations; organizational capacity; and a diversity in recipient type (e.g., health department, higher education institute, health system, etc.), partner type, geography (rural/urban/suburban), location (city/county/state), population of focus, implementation setting, focus on health equity/SDOH; participation in Evaluability Assessment; and participation in multiple CDC cooperative agreement (for cross-program coverage). The number of selected recipients per cooperative agreement is intended to be proportional to the number of awards, however the sample size will depend on the number of recipients who respond to the Call for Interest. The cost study includes a Cost Inventory Tool and semi-structured interviews that will be completed by both recipients and their select partners. Due to the nature of the call for interest process, we anticipate a near 100% response rate for the Cost Study.

Recipient Cost Study

Recipient Cost Inventory Tool (Attachment 5a): Approximately 1-2 staff per recipient will complete the cost inventory tool. The Comprehensive Evaluation Team will collaborate with recipient program managers/team leads to identify and connect with program staff who are knowledgeable about the program budget. These staff may include the Project Investigator, Program Manager or other identified recipient staff with comprehensive understanding of budget/staffing structures.

Recipient Interviews (Attachment 5c): The Comprehensive Evaluation Team will conduct in-depth, semi-structured interviews with up to two key informants at each recipient site for a maximum total of 110 respondents. Interview participants will be recipient staff who completed the cost study inventory tool as well as other identified staff with a comprehensive understanding of budget and staffing structure).

Document Reviews: The Comprehensive Evaluation Team will aggregate and review relevant program documents, such as recipient work plans and recipient budget narratives, as an input for the Cost Inventory Tool.

Partner Cost Study

Once recipients are selected for participation, the recipients will be asked to identify partners who are interested in participating in the Cost Study. If more than the maximum sample size of partners are interested, then the Comprehensive Evaluation Team will work with the recipient program manager/team lead to create a sampling frame of the recipients’ partners. Up to three partner organizations per recipient will be selected using a purposive sampling approach that applies the following prioritization criteria: implementation capacity, history of collaboration with recipient organization, implementation setting, type of organization (e.g., large hospital, rural health center, social services, community organizations. etc.), and willingness of the site to participate.

Partner Cost Inventory Tool (Attachment 5b): One staff member per participating partner organization will complete the cost inventory tool, for a total of up to 165 respondents. Respondents may include Program Managers, Program Coordinators, or other identified staff with an understanding of implementation activities and budget/staffing structures.

Partner Interviews (Attachment 5d): The Comprehensive Evaluation Team will also conduct in-depth, semi-structured interviews with recipients’ partners that were selected for the Cost Study. Interviewees will be the staff member who completed the cost study inventory tool or program managers, program coordinators, or other identified staff with an understanding of implementation activities and budget and staffing structures.

B2. Procedures for the Collection of Information

Information will be collected from recipients on an annual basis, at most. Data collection procedures vary slightly for each component of the evaluation, and those methods are described below.

Evaluability Assessments

The Evaluability Assessment nomination process and interviews with recipient and partners are planned for Year 2 of the cooperative agreements. For The National CVH Program and The Innovative CVH Program, the Evaluability Assessment for LCs are also planned to take place in Year 2. Data collection will include primary data (virtual semi-structured key informant interviews) and secondary data (document review).

The Comprehensive Evaluation Team will email a call for nominations to each recipient. The email will describe the purpose and components of the Evaluability Assessment and how findings will be used to inform future evaluation; explain how the recipients may be selected to participate in the Exploratory Assessments; share the benefits of participating; provide details about the nomination and selection process; clarify the expected time that the interviews will take to complete; and provide contact information for the evaluation team. The Comprehensive Evaluation Team will instruct recipients to complete the self-nomination form included in the email if they are interested in participating in the Evaluability Assessment in Year 2 and the Exploratory Assessment in Year 4, should they be chosen. The Comprehensive Evaluation Team will collaborate with the selected recipients to identify partner organizations, Learning Collaborative members, and individuals within each organization for invitation to the cost study.

For both recipient and partner site data collection efforts, the Comprehensive Evaluation Team will send an invitation email to identified candidates. The invitation email will further explain the logistics of the Evaluability Assessment and how insights gained will be used; specify that participation is voluntary; describe how individual-level responses will be safeguarded; clarify the expected time that the interviews will take to complete; request submission of relevant program documents for document review, provide details to begin scheduling interviews; and provide contact information for the Comprehensive Evaluation Team. Once they accept the invitation to participate, recipient and partner site staff members will receive a confirmation email that contains the interview date and time, Zoom link, and an attachment of the data collection instrument. The confirmation and Zoom link will include a meeting password and the interviewer and notetaker will monitor and control who joins the video conference to ensure the security and confidentiality of the participants.

The Comprehensive Evaluation Team will review relevant program documents before interviews in Year 2 to prepare for interviews. Document review will help familiarize the Comprehensive Evaluation Team with the sites and inform the development of logic model and probes for semi-structured interview guides. Other documents may be obtained during and after the virtual interviews.

Semi-structured key informant interviews with recipient and partner staff members will be conducted virtually and will last no more than 90 minutes. Interviews with Learning Collaborative members will also be conducted virtually and will last no more than 60 minutes. A primary interviewer from the Comprehensive Evaluation Team will lead all interviewers and will be supported by a notetaker. Prior to the interview, participants will be asked to provide verbal consent for participating in the interview. Additionally, all interviews will be recorded for transcription purposes with participant’s consent.

As interviews are completed, the Comprehensive Evaluation Team will send participants a follow up email thanking them for their participation, sharing the anticipated timeline for data analysis and results, and letting them know whom to contact with further questions.

Exploratory Assessments

The Exploratory Assessment with recipient and partner staff members is planned for Year 4 of the cooperative agreements. For The National CVH Program and The Innovative CVH Program, the Exploratory Assessment for LCs are also planned for Year 4. Data collection will include primary data (virtual semi-structured key informant interviews) and secondary data (document review).

For both recipient and partner site data collection efforts, the Comprehensive Evaluation Team will send an invitation email to identified candidates. The invitation email will further explain the logistics of the Exploratory Assessment and how insights gained will be used; specify that participation is voluntary; describe how individual-level responses will be safeguarded; clarify the expected time that the interviews will take to complete; request submission of new program documents (developed since the Evaluability Assessment data collection) provide details to begin scheduling interviews; and provide contact information for the Comprehensive Evaluation Team. Once they accept the invitation to participate, recipient, LC, and partner site staff members will receive a confirmation email that contains the interview date and time, Zoom link, and an attachment of the data collection instrument. The confirmation and Zoom link will include a meeting password and the interviewer and notetaker will monitor and control who joins the video conference to ensure the security and confidentiality of the participants.

The Comprehensive Evaluation Team will aggregate and review program documents developed after the Evaluability Assessment before interviews in Year 4 to learn about new program developments (if any) and to inform probes for the semi-structured interviews. All interviews will be conducted virtually. Recipient and partner interviews will last no more than 90 minutes and Learning Collaborative interviews will last no more than 60 minutes. A primary interviewer from the Comprehensive Evaluation Team will lead all interviewers and will be supported by a notetaker. Prior to the interview, participants will be asked to provide verbal consent for participating in the interview. Additionally, all interviews will be recorded for transcription purposes with participant’s consent.

As interviews are completed, participants will receive a follow up email thanking them for their participation, sharing the anticipated timeline for data analysis and results, and letting them know whom to contact with further questions.

Cost Study

The Cost Study with recipient and partner staff will take place during Year 3 and 4 of the cooperative agreements. Data collection will include secondary data (document reviews) and primary data (i.e., cost inventories and virtual key informant interviews).

The Comprehensive Evaluation Team will email a Call for Interest to each recipient. The email will describe the purpose and components of the Cost Study and how insights gained will be used; explain the benefits of participating; provide details about the nomination and selection process; clarify the expected time that the interviews will take to complete; and provide contact information for the evaluation team. Interested recipients will reply to the email with their intent to participate in the Cost Study. If needed, recipients from each cooperative agreement will be chosen for participation based on a purposive sampling approach and prioritization criteria. The Comprehensive Evaluation Team will collaborate with the selected recipients to identify partner organizations and individuals within each organization for invitation to the cost study.

For both recipient and partner site data collection efforts, the Comprehensive Evaluation Team will send an introduction email to identified candidates to invite them to participate in the Cost Study. The email will explain the logistics of the Cost Study and how insights gained will be used; specify that participation is voluntary; describe how individual-level responses will be safeguarded; clarify the expected time that the interviews and the cost inventory tool will take to complete; and will invite participants to attend a webinar that will describe the Cost Study process in more detail.

The Comprehensive Evaluation Team will host a webinar to provide guidance and expectations to recipients and their partners for completing the Cost Inventory Tool. During the webinar, we will recommend that each recipient and partner organization appoint a Cost Study Liaison to be responsible for gathering and reporting the cost data. The Comprehensive Evaluation Team will be available to the Liaison to answer questions and provide additional guidance, as needed. The webinar will be recorded and shared with participants who are unable to attend.

Both recipient and partner site cost study participants will input their cost data into the excel-based Cost Inventory Tool to document cost data related to the implementation of each strategy during program years 1 and 2. Prior to sending the Cost Inventory Tool to recipients, the Comprehensive Evaluation Team will review the budget and financial information reported by the recipient to CDC to pre-populate the tool. The types of documents for review will vary by site but may include recipient work plans, annual progress reports to identify activity progress and determine allocated budgets at the program level (including staffing, equipment, supplies, travel, and other costs). This will reduce data entry burden on recipients and provide them an opportunity to confirm or update pre-populated data and provide additional information that is not available in other reports. Partners do not provide budget or financial reports to CDC and therefore we will not pre-populate the partner template. Other documents may be obtained during and after the key informant interviews to provide additional context for cost data.

Data collected in the Cost Inventory Tool will include costs associated with personnel, equipment and materials, travel, subcontractors and consultants, and in-kind labor and non-labor inputs. Both direct and indirect (overhead) costs will be collected for analysis. All data will be self-reported by recipient, with guidance from the Comprehensive Evaluation Team. To assess costs of strategy implementation, the following cost data will be captured for recipients and select partners:

Personnel – Cost of staff providing direct services related to the Strategies. This will be calculated based on estimated level of effort (% time) dedicated to each Strategy and average salary by position.

Consultants/Subcontractors – Costs of consultants or subcontractors procured for implementation of Sub-strategies and activities.
Program Equipment, Supplies and Materials – Costs to prepare, print, disseminate marketing, communication, training, practice guideline, or other strategy specific materials.
Travel – Cost of travel to attend NOFO specific meetings or conferences; or to meet with partners/ sites.
Indirect/Overhead costs - Indirect costs are related to secondary costs incurred for running programs (e.g., support staff time, general office supplies, staff management/ supervision, etc.). These will include both capital (one-time expenditures) and recurring (annual) expenditures accrued in the sample year for each Strategy.
In-kind Contribution – Labor or non-labor contributions to the Strategies that are not funded through CDC funds.

The Comprehensive Evaluation Team will review the data provided in the cost study tool prior to the semi-structured key informant interviews to identify areas where the Comprehensive Evaluation Team need clarification and to prepare for the interview discussion.

Key informant interviews will be conducted after Cost Inventory Tool data collection to provide additional context to collected cost data, explore funding models and cost of sustainability and scale-up, identify cost facilitators and barriers, and to connect costs to outcomes. The Comprehensive Evaluation Team will send an invitation email to identified candidates. The invitation email will further explain the components of the Evaluability Assessment and how insights gained will be used; specify that participation is voluntary; describe how individual-level responses will be safeguarded; clarify the expected time that the interviews will take to complete; provide details to begin scheduling interviews; and provide contact information for the Comprehensive Evaluation Team. Once the respondent accepts the invitation to participate, identified staff members will receive a confirmation email that contains the interview date and time, Zoom link, and an attachment of the data collection instrument. The confirmation and Zoom link will include a meeting password and the interviewer and notetaker will monitor and control who joins the video conference to ensure the security and confidentiality of the participants.

B3. Methods to Maximize Response Rates and Deal with No Response

While participation in all data collection activities for the Comprehensive Evaluation is voluntary, we anticipate a response rate close to 100% due to the use of self-nomination. A requirement of self-nomination includes interest and capacity for both recipients and partner organizations in participating in the data collection efforts. To maximize response rates, The Comprehensive Evaluation team will use personalized email invitations to participate in the key informant interviews that will include information about how their organization was selected and how the information will be used to identify and share promising practices for CVD management and prevention. We will work closely with CDC Project Officers and or Evaluation Technical Assistance providers to engage recipient sites. CDC Project Officers and or Evaluation Technical Assistance providers have a close relationship with recipients and will provide an overview of the comprehensive evaluation and introduce our team to recipients, prior to our team reaching out to request nominations. Similarly, we will work closely with participating recipients who will describe the comprehensive evaluation to their partners and introduce the Comprehensive Evaluation Team before our team invites the partners to participate. We will work the recipient program managers to identify key recipient staff, partner organizations, and partner staff to ensure that the Comprehensive Evaluation Team is reaching out to individuals who are appropriately involved and invested in the cooperative agreements.

We will work with all interview respondents to identify a day and time that is convenient for their interview. Five business days in advance of an interview, participants will receive a reminder e-mail indicating the upcoming time and date of the interviews. If conflicts arise, we will work with the respondent to reschedule for a more convenient time.

To promote a higher response rate for the Cost Inventory Tool, the Comprehensive Evaluation Team will provide hands-on training and technical assistance as needed. The team will host a webinar to provide detailed instructions and will be available to respondents to assist with data entry questions.

B4. Tests of Procedures or Methods to be Undertaken

The Comprehensive Evaluation Team is convening voluntary internal and external advisory groups, or Evaluation Planning Groups (EPGs) to advise on the Comprehensive Evaluation planning, implementation, and use of findings. The internal advisory group, referred to as CDC EPG, is comprised of evaluation subject matter experts and DHDSP leadership at CDC. The external advisory groups, referred to as R-EPGs, exist for each cooperative agreement. The R-EPGs include representatives from 1) The National CVH Program, 2) The Innovative CVH Program, and 3) WISEWOMAN recipient teams.

The Applied Research and Effectiveness Branch (AREB) within DHDSP provided feedback on all primary data collection instruments (i.e., nomination form, interview guides, and cost inventory tools) for the Evaluability Assessment, Exploratory Assessment, and Cost Study Feedback was used to refine questions as needed, avoid duplicative areas, clarify question wording, ensure accurate programming and skip patterns, and promote our ability to conduct cross-program analysis.

The EPGs and a health economist within DHDSP provided feedback on feasibility and potential limitations for the Cost Inventory Tool. Feedback was used to determine sample size, refine the Inventory Tool, avoid areas duplicative of other data collection efforts, clarify cost categories, questions, and instructions, and establish the estimated time required to complete the tool.

The Cost Inventory Tool was adapted from previously developed cost tools, leveraging development and lessons learned from the DP18-1815 Resource Use and Cost Inventory Tool and the DP19-1907 Cost Tool. The Cost Inventory Tool may be pilot tested with up to two recipients and their partner sites to assess its ability to provide requested data and identify approaches to minimize burden. During the pilot test, the clarity of the instrument, usability of the Excel workbook, and accuracy of the data entered will also be assessed. The tool will be streamlined to only include the most important questions to inform the relevant assessment questions; therefore, no extraneous information will be collected.

The interview guides will not undergo any testing prior to data collection as they are semi-structured in nature so that interviewer can tailor questions to specific recipient sites and adjust the data collection instrument to meet the time limits of the interview.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

CDC staff from DHDSP and staff from Deloitte were consulted about the substantive, methodological, and statistical aspects of the study. Their recommendations were incorporated into the study design and instruments on an ongoing basis. Table B.5A lists the individuals consulted.

Staff members from Deloitte designed the data collection tools with input from CDC. Deloitte staff will also be responsible for overseeing and executing the collection of data with input from CDC staff. The data analysis will be led by Deloitte in close consultation with CDC staff throughout the process. Marla Vaughn is responsible for receiving and approving contract deliverables.

Table B.5-A Individuals Consulted on Methods and Analysis

Marla Vaughn, Lead Health Scientist