Attachment
B2: Interview Protocols for MH Portfolio Awardees
OMB No. 0906-XXXX
Exp. Date
XX/XX/20XX
Attachment B2. Maternal Health Portfolio Evaluation Interview Protocol for Awardees (Awardee Leadership, Evaluators, and Activity Implementation Staff)
Public Burden Statement: This is a new Information Collection Request (ICR) requesting approval to collect data for a portfolio-wide evaluation of Maternal Health (MH) programs funded by the Health Resources and Services Administration. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this information collection is 0906-XXXX and it is valid until XX/XX/202X. This information collection is voluntary. Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to HRSA Reports Clearance Officer, 5600 Fishers Lane, Room 14N136B, Rockville, Maryland, 20857 or [email protected].
The Evaluator will tailor each protocol to the awardee’s program prior to conducting the interview. Evaluators will determine, in collaboration with HRSA, which activities the awardee is implementing based on program documents. The activities will all correspond with data collection elements of the seven MH Portfolio strategies from the evaluation design, including: 1) Establishing and Strengthening Partnerships and State Capacity; 2) Workforce Training and Expanding the Maternal Health Workforce; 3) Providing Technical Assistance; 4) Improving Access and Coordinating Care; 5) Implementing Quality Improvement Initiatives, including Maternal Safety Bundles; 6) Implementing Telehealth; and 7) Creating and Disseminating Products.
Because there is such a wide range of activities being implemented across the MH Portfolio, HRSA has decided to limit evaluation at the activity level to seven specific activities. [Strategy: Implementing Clinical Quality Improvement Initiatives]
[Strategy: Improving Access and Coordination of Care]
[Strategy: Workforce Training and Expanding the Maternal Health Workforce]
[Strategy: Implementing Telehealth]
If the awardee is implementing any of these activities, they will be asked additional follow-up questions about these activities at various points throughout the interview (as indicated by a purple box). |
Introductory Script and Informed Consent (Verbal)
Good morning/afternoon. My name is [NAME] and I am a [TITLE] at [ORGANIZATION]. I am calling in reference to the HRSA Maternal Health Portfolio Evaluation, an evaluation of five HRSA programs including [AWARDEE’S GRANT PROGRAM].
If speaking with awardee respondent: Is this still a good time to conduct the interview?
If no: Can we schedule another time to conduct the interview that works for you?
If yes: Before we get started, I am going to read some information about the evaluation and your participation in the interview. It will only take a few minutes.
[ORGANIZATION] is conducting an evaluation for the Health Resources and Services Administration, or HRSA, about addressing maternal mortality. We are interested in speaking with you today to learn about barriers and facilitators to implementing your program, how your program is addressing health equity, as well as contextual factors and potential indicators for scaling and replicating your program’s strategies and activities.
This interview is being conducted for the evaluation of HRSA’s Maternal Health Portfolio Project. The portfolio includes five maternal health programs, including the [insert name of program] program. The overall focus of the maternal health portfolio evaluation include the implementation of awardees’ activities, opportunities for scaling and spreading effective program interventions, and the overall impact of the portfolio on maternal health outcomes.
Our discussion today will last about one hour. Information from this interview will be included in documents associated with the Maternal Health Portfolio evaluation for HRSA. While we will not use your name or any others in these reports, it may be possible to identify you through your position or through other details that you share in this interview. The information we gather will help HRSA to understand contextual factors and potential indicators for scaling and replicating your programs strategies and activities.
If you have questions about the study, please contact the Maternal Health Data Lead, Theresa Chapple-McGruder, 301-594-4421
Do you have any questions about the evaluation or your participation in the interview today?
Do you agree to participate in this interview?
If "Yes" then proceed.
If No: Thank you for your time. [Stop the interview].
My colleague [NAME] is also on the phone and will take notes during our conversation. We will also create an audio recording of the interview. We will use the recording to create a transcript to inform our report. The transcript will not include your name or contact information. We will delete the recording at the end of the project.
Do you agree to have this interview recorded?
If respondent says "yes" then proceed. BEGIN RECORDING.
If "no" then say: "That's fine. You may still participate in the interview. Please be patient as I take notes." DO NOT BEGIN RECORDING.
I’d first like to start off by asking you a few questions about your program.
Based on your application, the goals of the project are [list project goals]. Have any changes been made to the stated goals? If yes, please describe.
What is/are the target population(s) of your program? Prompts if respondent is unsure – ask for each program component:
Racial/ethnic group
Geographic area
Socio-economic status
Have there been any contextual factors, like policies or other state-level maternal mortality efforts or programs that have played a role in the implementation of the program? Prompts if respondent is unsure:
Pre-existing partnerships
State and national health policies
Maternal Mortality Review Committees (MMRC)
Title V
Number of hospital closures and OB closures state-wide
Who are the main partner organizations involved in program implementation? For each partner organization please describe their role.
Based on program documents, including your grant application and your program progress report, your program is working on the following activities [insert activities]. Is this correct? Have you made any changes?
We would like to know more about how your program is implementing these activities. In the next part of the interview, I am going to ask you a set of questions about each activity.
For your program efforts related to [ACTIVITY #1]:
What are the goals for this activity?
What are the main objectives associated with implementing this activity?
(Follow-up questions for activities related to Providing Technical Assistance only) How does the technical assistance you provide address health equity? Telehealth? Maternal safety bundles?
The following set of questions will ONLY be asked for the following activities:
|
For the activity, please describe the implementation settings.
Prompt if respondent is unsure: Hospital, Community health center, Behavioral health center, Non-medical setting
Who is/are the target population(s) of the activity?
Prompt if respondent is unsure: Race/ethnicity, Geography, Socio-economic status, Education
Please describe the staff involved in implementing the activity.
Prompt if respondent is unsure: Administrative staff, Physicians, Medical Assistants, Health department staff, Policy makers, Implementing organization vs partner organization
Next, we are interested in the potential to replicate or scale this aspect of your program.
What are the unique or important characteristics of this activity, including approaches that are evidence-based, that would be essential to someone trying to replicate the program?
What resources, such as infrastructure, cost, space, materials, and technical capacity, have been used for implementing this activity? Please indicate whether the resource was from your organization or a partner organization.
What contextual factors have played a role in the implementation of this activity?
Prompt if respondent is unsure: Pre-existing partnerships, State and national health policies, Maternal Mortality Review Committees (MMRC), Title V, Number of hospital closures and OB closures state-wide
The Evaluator will repeat Section 1 of questions for each program activity.
|
I’d like learn a bit more about how your program is collecting, managing, and reporting data for program evaluation.
Could you please describe the current status of your evaluation design, data collection, data analysis, and data reporting?
What challenges have you encountered with data collection and how have you addressed them?
To what degree are your data collection activities being implemented on the intended time frame? If not on time, why not?
Now I’d like to find out about the barriers and facilitators you’ve encountered during the planning and start-up phases of the program.
To what degree is the program being implemented as intended? If there have been any changes, please describe.
Did you or the program as a whole experience any challenges during the planning and start-up phases of the program? If yes, please describe. Prompts if the respondent is unsure:
State or local policies, partnerships, or existing maternal health programs/initiatives (e.g., Title V, MMRC, Perinatal Quality Collaborative, Levels of Care Designations),
Leadership buy-in (e.g., awardee-level or activity-level),
Staffing (e.g., adequate number of staff, turnover, appropriately trained staff),
Access to required infrastructure,
Access to additional resources and support,
Technical assistance,
Target population factors (e.g., social determinants of health such as poverty rates, social norms, health literacy, transportation access, unemployment)
Factors outside of program control (e.g., contextual factors, national or regional events, COVID-19)
What did you or the program as a whole find to be helpful during the planning and start-up phases of the program?
Now I’d like to find out about how your program is addressing maternal health equity.
How does your program ensure that it meets the cultural and linguistic needs of your target population(s)?
CLAS Reference for Interviewer to provide as prompts if the respondent is unsure:
Principal Standard
Provide effective, equitable, understandable, and respectful quality care and services that are responsive to diverse cultural health beliefs and practices, preferred languages, health literacy, and other communication needs.
Governance, Leadership and Workforce
Advance and sustain organizational governance and leadership that promotes CLAS and health equity through policy, practices, and allocated resources.
Recruit, promote, and support a culturally and linguistically diverse governance, leadership, and workforce that are responsive to the population in the service area.
Educate and train governance, leadership, and workforce in culturally and linguistically appropriate policies and practices on an ongoing basis.
Communication and Language Assistance
Offer language assistance to individuals who have limited English proficiency and/or other communication needs, at no cost to them, to facilitate timely access to all health care and services.
Inform all individuals of the availability of language assistance services clearly and in their preferred language, verbally and in writing.
Ensure the competence of individuals providing language assistance, recognizing that the use of untrained individuals and/or minors as interpreters should be avoided.
Provide easy-to-understand print and multimedia materials and signage in the languages commonly used by the populations in the service area.
Engagement, Continuous Improvement, and Accountability
Establish culturally and linguistically appropriate goals, policies, and management accountability, and infuse them throughout the organization's planning and operations.
Conduct ongoing assessments of the organization's CLAS-related activities and integrate CLAS-related measures into measurement and continuous quality improvement activities.
Collect and maintain accurate and reliable demographic data to monitor and evaluate the impact of CLAS on health equity and outcomes and to inform service delivery.
Conduct regular assessments of community health assets and needs and use the results to plan and implement services that respond to the cultural and linguistic diversity of populations in the service area.
Partner with the community to design, implement, and evaluate policies, practices, and services to ensure cultural and linguistic appropriateness.
Create conflict and grievance resolution processes that are culturally and linguistically appropriate to identify, prevent, and resolve conflicts or complaints.
Communicate the organization's progress in implementing and sustaining CLAS to all stakeholders, constituents, and the general public.
Is your program addressing bias among professionals and healthcare providers? If yes, how?
Is your program reducing barriers to accessing care among target populations? If yes, how? Prompts if the respondent is unsure: Race/ethnicity, Geography, Socio-economic status, Education
How is your program implementing quality improvement initiatives to improve quality of care among target populations?
Prompts if the respondent is unsure: Race/ethnicity, Geography, Socio-economic status, Education
I’d first like to start off by asking you about updates and program progress over the past year.
Have there been any changes to your program plan over the past year? If yes, please describe any changes to the program – including specific activities – over the past year. Probes/re-worded question if the respondent is unsure: To what degree is the program being implemented as intended? If there have been any changes, please describe
Please describe the program’s progress towards meeting each goal.
Last year we discussed the activities that your program is implementing. Based on program documents, including your grant application and your program progress report, and what you just described, your program is working on the following activities [insert activities]. Is this right? Have there been any changes or activities I missed?
I would like to know more about how your program has implemented these activities over the past year. In the next part of the interview, I am going to ask you a set of questions about each activity.
For your program efforts related to [ACTIVITY #1]:
What were the main objectives associated with implementing this activity in the past year?
(Follow-up questions for Providing Technical Assistance only) How does the technical assistance you provide address health equity? Telehealth? Maternal safety bundles?
The following set of questions will ONLY be asked for the following activities:
|
For the activity, please describe the implementation settings.
Prompt if respondent is unsure: Hospital, Community health center, Behavioral health center, Non-medical setting
Who is/are the target population(s) of the activity?
Prompt if respondent is unsure: Race/ethnicity, Geography, Socio-economic status, Education, Providers
Please describe the staff involved in implementing the activity, including project staff and partner staff.
Prompt if respondent is unsure: Administrative staff, Physicians, Medical Assistants, Health department staff, Policy makers, Implementing organization vs partner organization
Next, we are interested in asking about the potential to replicate or scale this aspect of your program.
What are the unique or important characteristics of this activity, including approaches that are evidence-based, that would be essential to someone trying to replicate the program?
What contextual factors have played a role in the implementation of this activity?
Prompt if respondent is unsure: Pre-existing partnerships, State and national health policies, Maternal Mortality Review Committees (MMRC), Title V, Number of hospital closures and OB closures state-wide
To what degree were changes made to the implementation of the activities based on quality improvement/evaluation efforts?
The Evaluator will repeat Section 1 of questions for each program activity.
|
I’d like learn a bit more about how your program is collecting, managing, and reporting data for program evaluation.
Could you please describe the current status of your evaluation design, data collection, data analysis, and data reporting?
What challenges have you encountered with data collection and how have you addressed them?
To what degree are your data collection activities being implemented on the intended time frame? If not on time, why not?
Now I’d like to find out about your program’s experiences receiving technical assistance.
What technical assistance did you receive from the Maternal Health Learning and Innovation Center (MHLIC) over the past year?
How would you describe the quality, knowledge, and effectiveness of the trainer?
How would you describe the quality, relevance, and timeliness of the content?
Did you have to consult additional resources to meet your technical assistance need?
To what extent did the technical assistance contribute to the program’s ability to meet program goals?
What technical assistance did you receive from AIM over the past year?
How would you describe the quality, knowledge, and effectiveness of the trainer?
How would you describe the quality, relevance, and timeliness of the content?
Did you have to consult additional resources to meet your technical assistance need?
To what extent did the technical assistance contribute to the program’s ability to meet program goals?
Now I’d like to find out about the barriers and facilitators you’ve encountered in the past year while planning and implementing the program.
Did you or the program as a whole experience any challenges during the past year? If yes, please describe.
More specifically, have you had any challenges with any of the following:
State or local policies, partnerships, or existing maternal health programs/initiatives (e.g., Title V, MMRC, Perinatal Quality Collaborative, Levels of Care Designations),
Leadership buy-in (e.g., awardee-level or activity-level),
Staffing (e.g., adequate number of staff, turnover, appropriately trained staff), Access to required infrastructure,
Access to additional resources and support,
Technical assistance (if not discussed above)
Target population factors (e.g., social determinants of health such as poverty rates, social norms, health literacy, transportation access, unemployment)
Factors outside of program control (e.g., contextual factors, national or regional events, COVID-19)
What did you or the program as a whole find to be helpful during the past year of program implementation in terms of making progress towards your project goals and objectives?
Now I’d like to find out about how your program is addressing maternal health equity.
How does your program ensure that strategies and activities are meeting the cultural and linguistic needs of your target population(s)?
CLAS Reference for Interviewer to provide as prompts if the respondent is unsure:
Principal Standard
Provide effective, equitable, understandable, and respectful quality care and services that are responsive to diverse cultural health beliefs and practices, preferred languages, health literacy, and other communication needs.
Governance, Leadership and Workforce
Advance and sustain organizational governance and leadership that promotes CLAS and health equity through policy, practices, and allocated resources.
Recruit, promote, and support a culturally and linguistically diverse governance, leadership, and workforce that are responsive to the population in the service area.
Educate and train governance, leadership, and workforce in culturally and linguistically appropriate policies and practices on an ongoing basis.
Communication and Language Assistance
Offer language assistance to individuals who have limited English proficiency and/or other communication needs, at no cost to them, to facilitate timely access to all health care and services.
Inform all individuals of the availability of language assistance services clearly and in their preferred language, verbally and in writing.
Ensure the competence of individuals providing language assistance, recognizing that the use of untrained individuals and/or minors as interpreters should be avoided.
Provide easy-to-understand print and multimedia materials and signage in the languages commonly used by the populations in the service area.
Engagement, Continuous Improvement, and Accountability
Establish culturally and linguistically appropriate goals, policies, and management accountability, and infuse them throughout the organization's planning and operations.
Conduct ongoing assessments of the organization's CLAS-related activities and integrate CLAS-related measures into measurement and continuous quality improvement activities.
Collect and maintain accurate and reliable demographic data to monitor and evaluate the impact of CLAS on health equity and outcomes and to inform service delivery.
Conduct regular assessments of community health assets and needs and use the results to plan and implement services that respond to the cultural and linguistic diversity of populations in the service area.
Partner with the community to design, implement, and evaluate policies, practices, and services to ensure cultural and linguistic appropriateness.
Create conflict and grievance resolution processes that are culturally and linguistically appropriate to identify, prevent, and resolve conflicts or complaints.
Communicate the organization's progress in implementing and sustaining CLAS to all stakeholders, constituents, and the general public.
Is your program addressing bias among professionals and healthcare providers? If yes, how?
Is your program reducing barriers to accessing care among target populations? If yes, how? Prompts if the respondent is unsure:
Race/ethnicity
Geography
Socio-economic status
Education
How is your program implementing quality improvement initiatives to improve quality of care among target populations? Prompts if the respondent is unsure:
Race/ethnicity
Geography
Socio-economic status
Education
Have there been any changes to your program plan over the past year? If yes, please describe any changes to the program – including specific activities – over the past year. Probes/re-worded question if the respondent is unsure: To what degree is the program being implemented as intended? If there have been any changes, please describe
Have there been any major changes that changed the course of your work throughout the program?
Please describe the program’s progress towards meeting each goal.
Last year we discussed the activities that your program is implementing. Based on program documents, including your grant application and your program progress report, and what you just described, your program is working on the following activities [insert activities]. Is this right? Have there been any changes or activities I missed?
I would like to know more about how your program has implemented these activities over the past year. In the next part of the interview, I am going to ask you a set of questions about each activity.
For your program efforts related to [ACTIVITY #1]:
What were the main objectives associated with implementing this activity in the past year?
(Follow-up questions for Providing Technical Assistance only) How does the technical assistance you provide address health equity? Telehealth? Maternal safety bundles?
The following set of questions will ONLY be asked for the following activities:
|
For the activity, please describe the implementation settings.
Prompt if respondent is unsure: Hospital, Community health center, Behavioral health center, Non-medical setting
Who is/are the target population(s) of the activity?
Prompt if respondent is unsure: Race/ethnicity, Geography, Socio-economic status, Education, Providers
Please describe the staff involved in implementing the activity, including project staff and partner staff.
Prompt if respondent is unsure: Administrative staff, Physicians, Medical Assistants, Health department staff, Policy makers, Implementing organization vs partner organization
Next, we are interested in asking about the potential to replicate or scale this aspect of your program.
What are the unique or important characteristics of this activity, including approaches that are evidence-based, that would be essential to someone trying to replicate the program?
What contextual factors have played a role in the implementation of this activity?
Prompt if respondent is unsure: Pre-existing partnerships, State and national health policies, Maternal Mortality Review Committees (MMRC), Title V, Number of hospital closures and OB closures state-wide
To what degree were changes made to the implementation of the activities based on quality improvement/evaluation efforts?
The Evaluator will repeat Section 1 of questions for each program activity.
|
Now I’d like learn a bit more about how your program is collecting, managing, and reporting data for program evaluation.
Could you please describe the current status of your evaluation design, data collection, data analysis, and data reporting?
What challenges have you encountered with data collection and how have you addressed them?
Do you have any preliminary findings that you can share with us?
Now I’d like to find out about the barriers and facilitators you’ve encountered in the past year while planning and implementing the program..
Did you or the program as a whole experience any challenges during the past year? Over the course of the program? If yes, please describe.
More specifically, have you had any challenges with any of the following:
State or local policies, partnerships, or existing maternal health programs/initiatives (e.g., Title V, MMRC, Perinatal Quality Collaborative, Levels of Care Designations),
Leadership buy-in (e.g., awardee-level or activity-level),
Staffing (e.g., adequate number of staff, turnover, appropriately trained staff), Access to required infrastructure,
Access to additional resources and support,
Technical assistance (if not discussed above)
Target population factors (e.g., social determinants of health such as poverty rates, social norms, health literacy, transportation access, unemployment)
Factors outside of program control (e.g., contextual factors, national or regional events, COVID-19)
What did you or the program as a whole find to be helpful during the past year of program implementation in terms of making progress towards your project goals and objectives? Over the course of the program?
Please describe any lessons learned from implementing the [NAME OF AWARDEE’S PROGRAM]. Prompt if respondent is unsure: Overall lessons learned
Lessons learned for each activity
Now I’d like to find out about your program’s experiences receiving technical assistance.
What technical assistance did you receive from the Maternal Health Learning and Innovation Center (MHLIC) over the past year?
How would you describe the quality, knowledge, and effectiveness of the trainer? Did you find the TA enjoyable?
How would you describe the quality, relevance, and timeliness of the content? Were the materials useful? Easy to use? Difficult to use?
Did you have to consult additional resources to meet your technical assistance need?
To what extent did the technical assistance contribute to the program’s ability to meet program goals?
To what extent did the technical assistance impact your program’s organizational processes?
What technical assistance did you receive from AIM over the past year?
How would you describe the quality, knowledge, and effectiveness of the trainer? Did you find the TA enjoyable?
How would you describe the quality, relevance, and timeliness of the content? Were the materials useful? Easy to use? Difficult to use?
Did you have to consult additional resources to meet your technical assistance need?
To what extent did the technical assistance contribute to the program’s ability to meet program goals?
To what extent did the technical assistance impact your program’s organizational processes?
Was there any TA that you wish you would have received to help you implement your program?
Was there any TA that you requested but didn’t receive?
Do you have any feedback about how the TA you did receive could have been more effective?
Is there anything else that you want to share about the TA you received or did not receive throughout your program?
Before we wrap up, I’d like learn a bit more about how your program or components of your program might be scaled up or replicated in other communities. (The Evaluator can refer back to the activities discussed earlier in the interview to prompt discussion here).
Which parts of your program do you think could be successfully scaled? Why? By scaled we mean, implemented in a larger setting or with a larger population. Prompt if respondent is unsure:
Observed outcomes
Stakeholder buy-in/satisfaction
What conditions would be necessary for this part of your program to be successfully scaled? Prompt if respondent is unsure:
Financial resources required
Staff roles required
Activity processes
Political will or policy support required
Infrastructure required
Resources needed by patients (e.g., transportation)
Which parts of your program could be successfully implemented in other contexts? Why? Prompt if respondent is unsure:
Observed outcomes
Stakeholder buy-in/satisfaction
What conditions would be necessary for this part of your program to be successfully implemented in other contexts? Prompt if respondent is unsure:
Financial resources required
Staff roles required
Activity processes
Political will or policy support required
Infrastructure required
Resources needed by patients (e.g., transportation)
Are there parts of your program that would be challenging or too difficult to scale or spread (implement in other contexts)? Why? Prompt if respondent is unsure:
Observed outcomes
Stakeholder buy-in/satisfaction
Financial resources required
Staff roles required
Activity processes
Political will or policy support required
Infrastructure required
Resources needed by patients (e.g., transportation)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Emily Phillips |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |