0920-24FI In-Depth Assessment – Evaluability Assessment Recipient

Form Approved

OMB No. 0920-XXXX

Exp. Date xx/xx/20xx

In-Depth Assessment – Evaluability Assessment Partner Interview Guide – Clinical Quality Measure – WISEWOMAN

Note: Public reporting burden of this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; ATTN: PRA (0920-24XX)

Evaluability Assessment

Partner Interview Guide

Date of Interview
Interviewer
Notetaker
Organization Name
Organization Type
State
Organization City		Zip Code
Cooperative Agreement	​☐​ WISEWOMAN
Strategy	Strategy 1: Track and Monitor Clinical Measures
Interviewee Name(s)
Interviewee Role(s) or Title(s)

Introduction

Thank you for taking the time to participate in this interview. My name is <insert name> and I am with the Deloitte evaluation team. Our team is working with the CDC Division for Heart Disease and Stroke Prevention to evaluate the Well-Integrated Screening and Evaluation of WOMen Across the Nation (WISEWOMAN) program. As part of the CDC-led evaluation, we are conducting evaluability assessment interviews to provide detailed insight into how recipients and their partners are prioritizing populations of focus impacted by the high prevalence of cardiovascular disease through Strategy 1: Track and Monitor Clinical Measures. We hope to learn about the function, structure, goals, and activities of your program in today’s discussion. Additionally, the evaluability assessment will be used to identify recipients and partners with promising approaches, who will be invited to participate in an exploratory assessment during PY4.

Our team has drafted a logic model based on program materials that your team shared with us prior to this interview. We may refer to the draft logic model throughout the interview to facilitate discussion on program goals, activities, desired outcomes, and contextual factors.

This interview is expected to take no longer than 90 minutes. Please answer questions based on your own knowledge and experience. Remember, you are the expert and that there are no right or wrong answers. If at any time during the interview you are not clear about what we are asking, be sure to let me know. Your participation in this interview is completely voluntary. You may choose not to respond to questions at any time and it will not in any way impact the funding or technical assistance your organizations receive from CDC.

Steps will be taken to protect your privacy; no information that identifies you will be shared with anyone except our project staff. All information will be kept secure and any personally identifiable information will be removed when results are aggregated for analysis.

Do you consent to this interview?

☐ Yes

☐ No

With your permission, we would like to record this interview for transcription purposes. 

Do we have your permission to record?

​​☐​ Yes 

​​☐​ No 

 

Do you have any questions or concerns before we start the discussion? 

Background

Thank you again for participating in this interview. For reference, today’s interview we will be talking about Strategy 1, which is defined as:

Track and monitor clinical measures shown to improve health and wellness, health care quality, and identify patients at risk of and with CVD, particularly hypertension.

We will discuss the following sub-strategies under Strategy 1:

[Interviewer Note: Only describe the relevant sub-strategies for which the recipient organization has self-nominated.]

1A: Provide cardiovascular disease (CVD) risk assessment to under- and uninsured participants in the priority age range of 35- 64 years during the baseline, follow-up, and reassessment office visits, as appropriate.

1B: Use electronic health record (EHR) and health information technology (HIT) data to query, monitor, and track clinical and social services and support needs data for improved identification, management, referrals, treatment, and outcomes of those at risk of CVD, particularly hypertension.

1C: Use standardized procedures to identify social services and support needs of participants and monitor and assess the referral and utilization of those services, such as food assistance, transportation, housing, childcare, etc.

1D: Use metrics from program data to guide quality improvement activities, e.g., Plan Do Study Act (PDSA) cycles, participant and partner feedback, etc., to increase program enrollment, retention, and referrals to additional services.

1E: Use EHR, HIT or program data to identify health care disparities and address health outcomes within their WISEWOMAN population.

First, we would like to learn a little about you and your organization.

[Interviewer Note: Use the following question to confirm information learned from the nomination form and document review about interviewee’s organization.]

1. From the <nomination form, APR, work plan, EPMP, etc.>, we learned that your organization offers <programs and services> for <population >. Is this correct? Is there anything else you would like to add or clarify?

Probes:

• What types of services/programs, related to WISEWOMAN, does your organization offer??

• How long has your organization been offering these services/implementing these programs?

• Can you describe to me the different populations (i.e., race, ethnicity, socioeconomic status, age, etc.) that your organization typically serves?

[Interviewer Note: Use the following question to understand the interviewee’s role related to the nominated strategy/sub-strategies.]

2. What is your role and what are your specific responsibilities related to WISEWOMAN?

Probes:

• How long have you been working with <organization name>?

• How long have you been in this role?

• Can you tell me about your role in relation to <implementing EHRs/HIT, tracking and monitoring clinical measures within your organization, providing CVD risk assessments, using program data to guide quality improvement, use standardized procedures to identify social needs>?

Program Implementation

[Interviewer Note: Ask about each nominated sub-strategy for the Program Implementation questions.]

Next, we would like to discuss how your organization works with <name of recipient> to track and monitor clinical measures for hypertension control. We’re interested in learning more about the program goals, key activities, implementation strategy, and intended program reach.

[Interviewer Note: Use the following question to understand the implementation of the nominated strategy/sub-strategies. Confirm what we’ve learned from the document review and nomination form. Tailor the language based on how partner refers to their program and activities rather than using NOFO specific language]

3. According to the <organization’s program materials, recipient-led evaluation deliverables, nomination form>, your program approach related to <EHRs/HIT and CQM> is <description of program. > Can you tell us more about the key activities and core components of <program name>? Describe things like the types of interventions being implemented, how it is implemented, and in what settings.

Probes:

• Where is the intervention implemented?

• What EHR systems does your organization use?

• 1A: What program activities are being implemented to increase the number of CVD risk assessments to under- and uninsured participants between ages 35- 64 years?

• 1B: How does your program approach promote increased use of EHRs/HIT to query, track, and monitor measures for clinical and social services?

  • What are the steps involved?

• How are data on clinical and social services measures used to:

  • 1D: Guide quality improvement efforts?

• 1C: What interventions improve the identification of social services and support needs?

  • What standardized processes or tools are used? How do they support patient identification?

• 1C: What program activities promote the monitoring and assessment of referrals and utilization of social services?

  • What is the protocol for monitoring referrals and utilization of referrals?

  • What standardized processes or tools are used?

  • How are referral and utilization (i.e., referral enrollment and completion) data used to:

    • 1D: Guide quality improvement efforts?

• 1D: Describe the quality improvement activities (e.g., PDSA cycles) being implemented to increase program enrollment, retention, and referrals to services.

4. Now, that we understand more about <program name >, please describe the team that supports <EHR/HIT activities>.   

Probes:

• Does your organization have a dedicated quality improvement team? 

  • How big is this team, who is it comprised of? What are their roles? 

• What team is responsible for querying, tracking, and monitoring clinical and social service measures? What about referral and utilization data?

• Who is responsible for reporting out EHR/HIT data? Who reviews the data and/or receives this information?

5. What are the goals of the <program> and how will the <program> achieve these goals?

Probe:

• What do you hope to achieve through quality improvement and CQM-related activities?

6. What is the <program’s> population of focus? What demographics, patient characteristics, or geographies are you prioritizing?

Probes:

• What barriers do the population of focus face in terms of management and treatment of CVD? How do SDOH factors affect their CVD risk?

• What tools and resources have you used to understand or identify the health disparities in your population of focus?

7. How do your <EHRs/HIT activities> address health disparities related to hypertension?

Probes:

• How are quality improvement efforts tailored to the needs of your population of focus?

• How do EHR/HIT activities address the needs of your population?

• 1A: How do SDOH affect the patient uptake of CVD risk assessments? How does this affect your program approach?

• 1E: How are data extracted from EHR/HIT used to advance health equity and improve health outcomes?

8. How do you work with <name of recipient organization> to implement <CQM activities>?

Probes:

• In what ways does the <recipient organization name> support your work? For example, technical assistance, training opportunities, resources, networking, etc.

• What are the strengths in the partnership? What are the gaps?

9. What other partnerships are in place to support <CQM implementation>?

Probes:

• What types of organizations are you partnering with and in what ways do they support quality improvement?

• What has worked well and what hasn’t worked as well with your partnerships?

Now that we’ve learned more about your program approach from its goals and operations, we would like to learn more about the extent of program implementation, successes and challenges with implementation, and factors that may support or hinder activities.

10. Tell us more about your progress related to <name of program> and <EHRs/HIT and CQM activities>?

Probe:

• Tell us more about milestones and other achievements.

• What are some areas where you did not make as much progress as anticipated?

1. What are your future implementation plans? How will you continue your partnership with the <name of recipient organization>, if at all?

Probe:

• Please describe what you hope to accomplish in the next two years (i.e., by September 2026 – September 2027).

12. Can you tell us about the contextual factors that support or hinder activities related to <tracking and monitoring clinical measures>?

Probes:

• Describe external factors such as complementing or competing initiatives, additional funding sources, partnerships and collaborations, state policies, political/economic climate.

• Describe internal factors such as organizational policies, leadership buy-in, internal capacity, organizational culture.

13. What would you say are the strengths of your <program>?

Probe:

• What factors positively affect <CQM strategy/sub-strategy implementation> or help the <program> be successful?

1. What challenges have you experienced with <CQM strategy/sub-strategy implementation>?

Probes:

• How are these challenges addressed?

• What additional support, TA, or resources do you need to overcome these barriers?

15. What have been some key lessons learned from your experience partnering with <recipient and NOFO names> for <CQM implementation>?

Program Evaluation

[Interviewer Note: Ask about each nominated sub-strategy in the Program Evaluation questions.]

We would like to understand to what extent the <EHR/HIT and CQM activities> have been or are currently being evaluated. We are also interested in learning about your organization’s capacity to evaluate <program activities>.

1. Who is primarily responsible for tracking and reporting data to <name of recipient> related to <CQM implementation> for the WISEWOMAN program?

Probes:

• What is your role in data collection and monitoring and evaluation efforts?

• Are other members or organizations involved in data collection? Describe their roles and responsibilities.

2. What data do you or your organization collect to help monitor <program activities> and evaluate its success?

Probes:

• What types of metrics or indicators do you use to measure progress and monitor implementation?

  • [If the interviewee only mentions PMs] What about metrics other than the NOFO performance measures?

• [If the interviewee only mentions MDEs] What about metrics other than the MDE data?

• How is success measured?

• What outcomes do you track? What outcomes do you expect to have by September 2026 – September 2027 (Y4)?

  • [If the interviewee only mentions PMs] What about outcomes beyond the NOFO performance measures?

  • [If the interviewee only mentions MDEs] What about outcomes other than the MDE data?

[Interviewer Note: Only ask remaining questions if partner organization is involved with data collection and evaluation]

3. How do you collect the data needed for monitoring and reporting of <CQM>?

Probes:

• What data collection tools or instruments are used to track data (paper, electronic)?

  • [If not already provided] Request to see and get copies of data, data collection tools, or evaluation reports.

• Describe the process used to collect data.

• Describe how you use the data.

• Do you collect data at specified points over time (time series)? What length of time? How frequently?

What type of data related to health equity are collected and tracked? Probes:

• What SDOH data are collected?

• How will health disparities be measured and defined?

• What sort of methods or tools are used to measure health equity outcomes?

4. What, if any, barriers have you encountered with data collection or monitoring and reporting activities?

Probes:

• What challenges, if any, are there with collecting SDOH data?

• What barriers, if any, do you anticipate in data collection or reporting related to outcome or impact?

• What specific strategies have been used to overcome these barriers?

• What additional support or TA do you need from <name of recipient organization>? What about from the CDC?

• What have been some lessons learned from your experience with data collection and reporting related to <CQM> for the WISEWOMAN program?

5. What have you learned so far about your program from your monitoring and reporting efforts?

Probes:

• What findings can you share about implementation progress?

• What outcomes can you report at this point?

• How are data being used to make improvements?

6. What is the timeline for the next phase of data collection and reporting?

Probes:

• What are key events for data collection and reporting?

• Are there anticipated barriers for the next phase?

7. Do you share results with external audiences related to <CQM implementation> such as funders, partners, decision makers, constituents, or others?

Probes:

• What types of findings do you share? How do you share your findings?

• Do they use any of the information you provide? In what sorts of ways?

8. If <name of recipient> is selected to participate in an exploratory assessment of your program, to what extent would your site have the capacity to contribute to detailed data collection on outcomes and/or cost?

Probes:

• What kind of assistance do you think you would need?

• Are there any other important considerations we should know about your readiness/capacity to participate in an evaluation?

Closing

Lastly, what questions do you have for me? Is there anything else you’d like to share?

Thank you again for participating. This concludes our discussion about CQM implementation. If you have any additional questions, please feel free to contact the Comprehensive Evaluation Team, [email protected].