Part A, Attachment 4
Program Staff Interview Guide
Background
Purpose
Acumen, LLC and its partner, Westat, have been contracted by the Centers for Medicare & Medicaid Services (CMS) to conduct Phase III of the evaluation project entitled Evaluation of Evidence Based Community Wellness and Prevention Programs – Analysis of Beneficiary Readiness to Engage in Community Based Wellness and Prevention Activities and Prospective Evaluation Efforts. This study builds on the work completed under the previous two phases of the project and supports CMS’ goal to evaluate the impact of wellness programs on Medicare beneficiary health, utilization, and costs to determine whether broader Medicare beneficiary participation in wellness programs could lower future growth in Medicare costs. This study involves conducting a systematic and thorough evaluation integrating Medicare claims data with survey data, and qualitative information about program implementation, operations and costs in a prospective study design to develop the “business case” for wellness program participation by Medicare beneficiaries; the study also aims to assess demand for, or readiness to engage in, wellness programs in the general Medicare population. The qualitative component of the study includes a mixed-method data collection approach designed to gather specific information about the various wellness programs, including program implementation, operations and costs, as noted above. The qualitative analysis will be based on interviews with key wellness program staff.
This document is the draft protocol for capturing pertinent information about the selected programs conducted through in-depth, unstructured telephone interviews and subsequent site-visits with program staff. To prepare for the interviews, the evaluation contractor will request that the programs share readily available secondary material prior to the interview that describes the program, including information on policies and procedures, outreach and enrollment processes, training materials or program implementation toolkits, and other descriptive information. Secondary documents that address the topics in the in-depth discussion guide will be confirmed during the interviews and site visits.
Draft Interview Protocol
Introduction
Introduce self, purpose of the study, and goals for the interview. Answer any questions about the purpose and goals of the study.
Explain how the information provided during the interview will be used. Answer any other questions the program staff member may have about how this information will be used.
Cover informed consent issues, including the voluntary nature of participation, any risks, any benefits, and whom to contact with any questions.
If applicable, thank them for sending any written materials that they have provided in advance, and note that you will be exploring that information in more depth to get additional details tailored to the specific needs of the study.
Request permission to record the interview. If permission is granted, turn on tape recorder and have interviewee restate his/her permission.
Some questions may not need to be asked (e.g., Background, Program Outreach) if the program has previously sent material that addresses the question. Confirm or clarify understanding or outstanding questions from material, if needed.
Background
Describe your organization (e.g., structure, type, size, and any other pertinent characteristics) and its role in administering the wellness program.
Could you describe your role within the organization?
When the program was first implemented? Over what period of time?
Could you describe the goals and objectives of the program?
Program Outreach
What population does the program target?
How many participants does the program serve?
[PROBE FOR:]
Referrals from health care providers
Self-referrals
Direct-to-consumer marketing
Community-based organizations
How do participants enroll in the program? Are there certain criteria that must be met?
Do beneficiaries pay a fee to participate in the program? What effects, if any, do the fees have on enrollment, participation, and outcomes?
Have you noticed trends in the types of individuals who decide to participate in the program? If yes, please describe/explain.
What populations have you found are hardest to reach? What strategies have you used to encourage these individuals to participate? Which strategies worked or did not work?
What strategies do you use to promote awareness of your program or the value of the program to potential participants, referral sources, or others?
How would you describe the influence of caregivers on Medicare beneficiary participation? How frequently do caregivers participate in the program along with the beneficiary?
Program Description
How does the program work on a daily basis from the perspective of the participant?
What are the main elements or steps of the program?
When does the activity occur?
Where does the activity occur?
Was the program based on a theoretical framework or evidence-based information? If yes, please describe.
Does the program coordinate with other health care systems or health care programs?
[If applicable] How do you coordinate and communicate with other members of the health care team?
How do you coordinate with other wellness programs?
What infrastructure and non-financial resources (physical, cultural, technological, staff-based) are needed for program implementation?
SPECIFIC QUESTIONS FOR THE “CONVENING” ORGANIZATION
What are the “core” elements of the program? (i.e., how would you define fidelity to the model at a fundamental level?)
For what aspects of the program do you allow flexibility for local site adaptation and why?
SPECIFIC QUESTIONS FOR THE LOCAL SITE
How have you adapted the program to meet your local needs?
Monitoring and Outcomes
How do you evaluate or define the success of your program?
What outcomes do you track/have you tracked?
Do the outcomes vary by participant population?
Has the program benefited certain types of participants more than others? If yes, could you describe both the ‘benefit’ and the types of participants?
Have you experienced any unexpected results? If yes, why do you think these results occurred?
Has any evaluation of the program been conducted? Do you have any outcomes reports or data that you could share?
SPECIFIC QUESTIONS FOR THE “CONVENING” ORGANIZATION
What site-level information is reported back to you?
What data are collected, including frequency of submission, unit of reporting (e.g., participant, visit, etc.), and format of submission?
Are any reports provided back to the individual programs?
Have you noticed regional variation in outcomes?
Do you track or evaluate implementing sites’ fidelity to the model over time? If yes, what does this entail and how do you approach and approach fidelity? How is the information used?
Implementation
Can you describe how you get a program up and running?
How long does it typically take to implement the program, including enrolling a critical mass of participants?
Are there specific phases of implementation?
Which stakeholders are typically involved in implementation, and how did you get them engaged?
[ADDITIONAL PROBES:]
What is the message with each key stakeholder?
How and who delivers the message?
What lessons can you give about startup and implementation?
What resources and organizational support are necessary for implementation?
What challenges or barriers did you face in the implementation of the program? How were these overcome?
SPECIFIC QUESTIONS FOR THE “CONVENING” ORGANIZATION
How many sites have implemented the program (both current and historical)?
What do you do to prepare for implementing the program at multiple sites?
How does the implementation of the program vary across sites?
How do you decide what sites will implement the program?
How do you determine when a site is ready to implement the program?
What structures does a site need to have in place before implementing the program?
Are there formal guidelines for implementing the program?
Are there protocols or toolkits available to local sites implementing the program?
Do you offer any sort of technical assistance? If yes, please describe the nature and frequency of this support.
What support do you provide to sites that are just “rolling out” the program?
What ongoing support do you provide to established program sites?
Do you provide any forums, learning networks, or infrastructure for local sites to communicate with each other to share best practices or lessons learned?
Staffing
Can you describe the staff roles and responsibilities, including volunteers, who are involved with program implementation?
How many staff members (and volunteers) work on/for the program?
How many full-time equivalents are needed to run the program?
In general, what educational/experience background or qualifications are required for each staff member and for volunteers?
What training do staff members receive?
[PROBE FOR:]
Who provides the training
When it is given
Content of training
Length of training
Describe any other ongoing support that you give staff members.
Implementation and Operational Costs
How much does the program cost to implement (i.e., upfront costs)? [Probe to clarify what is being included in upfront costs]
Do you receive any external funding for implementation of the program? If yes, from where did the funding come? How much is this external funding?
Do you receive any in-kind support? If yes, please describe.
What are the annual operating costs? [Probe to clarify what is being included in annual costs]
Can you provide a breakdown of the major components of these costs?
[PROBE FOR COSTS RELATED TO:]
Labor (staffing/salaries)
Equipment
Licensing or franchise fees
Outreach materials and marketing
Staff training
Other administrative costs (e.g., overhead).
What is the estimated cost of providing the program for each participant?
How are the ongoing operating expenses for the program paid for?
Do you have any external support to help pay for ongoing operating expenses? If yes, from where does the funding come?
Are costs reimbursable? Do you receive any revenue from the providing the services of the program (other than the licensing fees, if applicable)?
Lessons Learned/Best Practices
What advice do you have for others who may want to implement this program?
Based on your experience, can you describe the implementation factors that contribute to successful implementation regarding:
Participant outreach and recruitment
Staff training and program development
Stakeholder engagement
Licensing agreement
Data reporting and collection
Can you describe implementation pitfalls that other programs should be aware of, and how you addressed these challenges?
Based on your experience, what would you say are best practices related to program operations and implementation?
Ongoing Operations and Sustainability
How has the program evolved over time?
Are there plans to change the program in the future? If yes, how?
What challenges or barriers have you faced in the ongoing implementation and operations of the program?
Which are the most pressing or difficult?
How were these overcome or what are you currently doing to address them?
How do you continuously improve the quality and efficiency of the program?
What do you do to differentiate your program from other similar wellness programs?
What are the keys to sustaining the program over time?
How do you maintain enthusiasm and avoid complacency among participants?
How do you keep key stakeholders engaged and ensure they remain supportive?
What are the most important factors required to maintain and sustain the program?
Conclusion
Are there any additional written materials or information (e.g., articles or newsletters about the program, reports, etc.) that would be helpful in addressing or contributing to the points discussed?
[IF YES, ASK FOR COPIES]
Could we contact you by e-mail or phone with any follow-up questions?
Thank the respondent for his/her time
Version: 9/5/2014
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Part A, Attachment 5 |
| Subject | In-depth interview guide with wellness program staff |
| Author | Westat |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |