Instrument 15: Program operator interview guide for the systems study

S upporting Statement for OMB Clearance Request

Instrument 15: HPOG 2.0 National Evaluation Program Operator Interview Guide for Systems Study

National and Tribal Evaluation of the 2^nd Generation of the Health Profession Opportunity Grants (HPOG)

0970-0462

April 2019

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Nicole Constance, Hilary Bruck, and Amelia Popham

Health Profession Opportunity Grants (HPOG) 2.0 Systems Study
Overview and Consent

As you know, [name of local HPOG program] is participating in a national evaluation of the Health Profession Opportunity Grants Program. The second round of grants was awarded in 2015 (referred to as HPOG 2.0) and is sponsored by the Administration for Children and Families within the U.S. Department of Health and Human Services. The evaluation is being conducted by Abt Associates, the Urban Institute, MEF Associates, NORC at the University of Chicago, and Insight Policy Research, Inc. The evaluation is studying HPOG‐funded education and training programs across the country and examining how they help low‐income individuals, including Temporary Assistance for Needy Families (TANF) recipients, secure well‐paying healthcare jobs.

As part of the HPOG evaluation, we are examining the service delivery systems in which HPOG programs operate. The results of this effort will help us to better understand how grants like HPOG can support and improve local and regional systems for healthcare training for low-income individuals. What we mean by the “service delivery system” includes the network of partners and stakeholders—including HPOG program operator, the lead organization directly responsible for the administration of an HPOG program—that deliver, support, or benefit from healthcare training offered in the local area or region and the activities that make the system function. Activities that support systems include collaborating, improving the quality and accessibility of training and support services, engaging industry and employers, using data for joint decision making, and increasing the scale and sustainability of training and services. The system also includes the broader training opportunities and support services available to low-income individuals and important contextual factors such as the local economy and federal, state, and local policy governing programs that affect the service delivery system.

To learn more about HPOG systems, we are interviewing program operators and partners from 12-16 HPOG programs. Our interview with you will take approximately 75 minutes to complete. We will ask you questions about your local service delivery system and the organizations that make up the system, both those involved and not involved in HPOG activities, and their roles in the system and with HPOG. We will also ask for your perspective on how the HPOG program may have influenced the system and how the system may have influenced HPOG implementation. We will also ask you to provide contact information for partner organizations so we can gain their perspectives on these issues.

Privacy Statement: Your answers will be kept private. Information you provide will not be shared with other staff at your program or organization. Only the evaluation team will have access to the information you provide through this interview. Names will not be listed in any published reports and comments will not be attributed to respondents, grantees, programs, or their partners. Instead, your information will be combined with information provided by other program and partner respondents. Your responses to these questions are completely voluntary. We hope you will choose to answer all of the questions in the interview, but you may choose to skip any question you do not feel comfortable answering or that does not apply to you.

OMB Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays an Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is 0970–0462. The time required to complete this collection of information is estimated to average 75 minutes, including the time to review instructions and provide respondent contact information for partners. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [email protected] and reference the OMB Control Number 0970–0462.

If you have any questions or concerns about participating in the interview, please let me know. Your verbal decision to participate in the interview will be considered as your consent. Do I have your consent to participate in this interview? [Optional: May we record this interview to ensure the accuracy of our notes?]

Instructions for interviewers: Prior to the interview, please review relevant information from the grantee, including applications, summary information from Participant Accomplishment and Grant Evaluation System (PAGES), Performance Progress Reports (PPRs), Evaluation Design Implementation Plans (EDIPs), first-round and (if available) second-round telephone interviews, and focus area site visits (where applicable, especially the employer engagement visits) so the lead interviewer can confirm this information and probe for additional information if needed. The team should also review information to understand whether a partner organization is involved in any other HPOG programs to ensure the team is clear on the HPOG program and interactions for the selected site. The junior interview team member should pre-fill the interview guide to help prepare for the telephone interview. If anything is unclear, the team may consult with the evaluation site liaisons for the HPOG program. Questions where information from these sources may be available are noted in blue below.

1. Respondent information

[Interviewers: Obtain information available from EDIPs and first-round and (if available) second-round telephone interviews, confirm, and ask if anything has changed since then.]

We want to confirm a few details about you before moving to our main questions.

a. Name, title, and organization

b. What is your role at [organization]?

c. What is your role for the [name of local HPOG program]?

d. Did you or your organization have any involvement in HPOG 1.0 (first round)?

e. Do you or your organization have any involvement in other HPOG 2.0 programs?

2. Description of local service delivery system and local context

We want to learn about the local service delivery system in which your organization operates. Let’s start with some basic aspects of the system about which you may have some knowledge.

a. We understand that the major programs for healthcare training currently available across your local service delivery system including [name of local HPOG program] are [NAME TRAINING PROGRAMS]. Is there anything missing from the list? [Interviewers: Populate with information from applications, first-round and (if available) second-round telephone interviews, EDIPs, PPRs, and PAGES. Confirm programs, then ask if we missed any.]

[Probe the following if not mentioned by respondent: programs primarily serving low-income individuals with challenges to education and employment]

b. What needs for healthcare training in your community has [name of local HPOG program] addressed?

[Probe the following if not mentioned by respondent:]

• Employers’ needs for healthcare training in the local area/region

• The degree to which TANF and low-income residents are able to access healthcare training

• External factors such as state and local policy or funding that affect how healthcare training is provided in your local area/region

c. Are there local initiatives in addition to HPOG to improve healthcare training and jobs, such as industry partnerships, career pathways initiatives or programs, and other grant programs that you know of? By career pathways, we mean the following: “a clear sequence, or pathway, of education coursework and/or training credentials” that are recognized by employers. What are they? How have they informed the design and implementation of [name of local HPOG program], if at all?

[Probe if not mentioned: How long have these initiatives existed? What are these initiatives trying to accomplish?]

d. How strong is the local/regional healthcare labor market? Growing or shrinking? How has it changed since the start of the HPOG 2.0 grant? Since the start of the HPOG 1.0 grant (if applicable)?

e. How well is the system responding to local labor market needs, for both workers and employers? Has the HPOG program contributed to improvements meeting employer needs? If yes, how so? If not, why not and what have been the challenges?

3. Description of the network of organizations in the system and their roles

We learned about how organizations partner with [name of local HPOG program] through some of our other evaluation activities. When we say “organizations,” we mean the range of entities such as education and training providers, workforce development agencies, nonprofit organizations (community- and faith-based), other government agencies, employers and industry organizations, and others that support healthcare training for low-income individuals. We want to hear more about the roles of the organizations and others in the system who contribute to healthcare training in the local area or region.

1. We have some information from earlier data collections for the HPOG Implementation Study about the HPOG lead organization and partners and the roles they play in [name of HPOG program]. Would you confirm that these are still the roles your partners are playing? [Interviewers: Populate this section with information on the HPOG partner roles from the first-round telephone interview.]

   1. From your perspective, which organizations have the most highly involved partners?

   2. From your perspective, which organizations have been less involved partners?

2. In addition to their role with HPOG, what else do these partners provide or how else do they support individuals in healthcare training? [Probe the following if not mentioned]

• Possible roles: provide employment services, provide education and training, offer supportive services, support employer HR needs, coordinate workforce strategies/policies, provide funding and resources to support system, improve job quality/access]

c. How do organizations across the system interact? In what ways do they work well together? What are the challenges to working together?

d. Which organizations do you consider highly involved in HPOG activities?

e. How do external factors such as funding and the economy affect collaboration?

f. To what degree are employers and industry involved in healthcare training for low-income individuals? In what ways? How do they interact with organizations in the system?

g. How have the roles of organizations in the system changed since the start of the HPOG 2.0 grant? Since the HPOG 1.0 grant (if applicable)? [Interviewers: Populate with information from second-round telephone interview data (if available) on whether partnerships expanded, diminished, or stayed the same since the first round of data collection.]

h. What do you see as the strengths of your system in providing healthcare training for low-income individuals?

i. What gaps do you see in the system? Are there weaknesses in supporting healthcare training for low-income residents?

4. System activities

Now we are going to ask about some of the broader set of systems activities such as collaborating, improving the quality and accessibility of training and support services, engaging industry and employers, sharing data for improving coordination, and sustaining healthcare training and services.

Collaboration

In this part of the interview, we will discuss the ways in which organizations in your local area/region collaborate to provide and support healthcare training for low-income individuals, both for HPOG and more broadly.

a. Does [name of organization] coordinate with other organizations in the system? If yes, what activities do you coordinate on? If no, why not and what are the challenges to doing so? [Move to the next section if answers “no”.] [Interviewers: Populate with information from the first-round telephone interviewers on collaboration across the system.]

b. How do organizations across your system currently collaborate or work together? Which organizations are most engaged in the collaboration? In what way do these organizations collaborate?

[Probe: Ask about collaboration related to access and quality of training, employer engagement, data sharing, and scale/sustainability of programs. Also, probe collaboration as part of [name of local HPOG program] and outside of HPOG/business as usual.]

c. Are there organizations who do not generally collaborate with others? Do you know why they don’t?

d. Is there a shared vision for or coordinated approach to collaboration across organizations or for specific initiatives, such as [name of local HPOG program]? If so, would you describe the vision or approach? What are the goals of this collaboration?

e. Which organization or organizations lead collaborative efforts?

[Probe for if not mentioned: Is there a workforce intermediary coordinating across organizations? An intermediary is an organization that serves as a coordinating body for workforce development activities in a local area or region.]

f. How did these collaborations begin (if you have that history)? What was the impetus?

[Probe: HPOG [1.0/2.0] or other grants or initiatives?]

g. Did organizations that had not previously collaborated become involved? In what ways?

h. Did organizations with existing partnerships strengthen these relationships? In what ways?

i. What have these collaborations achieved to date?

j. Overall, do you think [name of HPOG program] has improved collaboration across the system since the start of the HPOG 2.0 grant? Since the HPOG 1.0 grant (if applicable)? If yes, in what ways? If not, why?

k. How did partner organizations help [name of HPOG program] to achieve the HPOG program’s goals?

Quality and accessibility of healthcare training

We asked earlier about the ability of low-income individuals to access healthcare training. We want to discuss efforts by organizations in your system to improve the quality and accessibility of healthcare training. Typically, these efforts make healthcare training more visible and accessible to low-income individuals, ensure that healthcare training teaches the skills and offers credentials needed by employers, and supports participants with challenges to succeeding in education and employment.

a. Has [name of organization] participated in efforts to improve the quality and accessibility of healthcare training in your local or regional system? If yes, what efforts have [name of organization] participated in? As a part of [name of local HPOG program]? For other efforts? If not, why not and what are the challenges?

b. Have organizations across the system helped to improve the quality and accessibility of healthcare training? If yes, as a part of the HPOG program? For other efforts? If not, why not and what are the challenges? [Move to the next section if answers “no.”]

c. What have been the goals for improving quality and accessibility? As a part of [name of local HPOG program]? Other efforts? To what degree have these goals been achieved? What have been the challenges?

d. What role, if any, has the development and use of career pathways models played in increasing the quality and accessibility of healthcare training? As a part of [name of local HPOG program]? Other efforts?

e. Did the [name of local HPOG program] lead to any new or revised policies or dedicated funding sources for healthcare training? If so, what were they?

f. How has [name of local HPOG program] improved the quality and accessibility of healthcare training since the start of the HPOG 2.0 grant? Since the start of the HPOG 1.0 grant (if applicable)?

g. How have other organizations in the system helped [name of local HPOG program] improve the quality and accessibility of the HPOG-funded training?

Employer and industry engagement

We touched on this earlier but we want to ask some specific questions about how organizations in your system engage employers and industry to support healthcare training for low-income individuals.

[Interviewers: Populate with information from the first-round and (if available) second-round telephone interviews on collaboration across the system. Data from employer engagement site visits may also be relevant.]

a. Has [name of organization] been involved in employer and industry engagement? If yes, how [name of organization] been involved in these efforts? For [name of local HPOG program]? Other efforts? If not, why not and what are the challenges?

b. Have other organizations across the system coordinated employer and industry engagement activities? If yes, as a part of [name of local HPOG program]? Other efforts? If not, why not and what are the challenges? [Move to the next section if answers “no.” to both a. and b.]

c. What have been the goals for engaging employers and industry? For [name of local HPOG program]? Other efforts? To what degree have these goals been achieved? What have been the challenges?

d. How has [name of local HPOG program] helped to change employer and industry engagement to support healthcare training for low-income individuals since the start of the HPOG 2.0 grant? Since the start of the HPOG 1.0 grant (if applicable)?

[Probe for HPOG 1.0 involved respondents: Did the increased emphasis on employer and industry partnerships for the HPOG 2.0 grants change the intensity of activities and engagement of employers and industry? If so, what activities led to this change? If not, why not?]

e. How have other organizations in the system helped [name of local HPOG program] engage employers and industry?

Data sharing for improved coordination

Data sharing may be another aspect of systems efforts, where organizations collect, use, and share data to develop strategic plans and priorities, design and continuously improve programs and services, and track outcomes for workers and employers.

a. Do organizations across the system, including yours, share data on participants or on other aspects of program management to improve coordination? If not, why not and what are the challenges? [Move to the next section if answers “no”.]

b. If organizations are sharing data, what data has been shared across organizations? Where did the data come from? How has the data been used? For [name of local HPOG program]? Other efforts? What data was not able to be shared?

c. How has [organization] been involved in data sharing to improve coordination for healthcare training for low-income individuals?

d. How have organizations across the system been involved in data sharing to improve coordination? How have they coordinated data sharing across organizations? As a part of [name of local HPOG program]? Other efforts?

e. What have been the goals for data sharing? For [name of local HPOG program]? Other efforts? To what degree have these goals been achieved? What have been the challenges?

f. How has [name of local HPOG program] supported data sharing to improve coordination across the system? How has [name of local HPOG program] hindered data sharing across organizations?

g. How have other organizations in the system helped [name of local HPOG program] share data? How have other organizations in the system hindered data sharing?

Sustainability

Developing healthcare training that can be sustained over the long run can be an important system goal, especially for efforts like HPOG where the grant will eventually end. We want to learn how organizations in your system are planning to sustain healthcare training for low-income individuals for [name of local HPOG program] or as part of other efforts after the grants end.

[Interviewers: Populate with information from the first-round and (if available) second-round telephone interviews on sustainability. Focus site visits on sustainability may also be relevant.]

a. What are the strategies for sustaining healthcare training for low-income individuals after federal funding for [name of local HPOG program] ends? If there are no strategies, why not?

b. If there are strategies, how is [organization] involved in developing and implementing strategies for sustaining [name of local HPOG program] and other healthcare training?

c. How are organizations across your system coordinating to sustain [name of local HPOG program] and other healthcare training? As a part of [name of local HPOG program]? Other efforts?

d. From your perspective, what changes to the broader system have occurred that may help sustain [name of local HPOG program] after the grant ends?

e. What factors may contribute to [name of local HPOG program] ending or scaling back operations once the grant ends?

5. Wrap-up questions

You have provided a lot of useful perspectives on your local and regional system and how it has supported healthcare training for low-income individuals. We have a few final questions for you to consider.

a. What do you think are the most important contributions [name of local HPOG program] has made to the local service delivery system?

b. Where do the gaps in healthcare training and support services for the system remain? Once the grant ends, will these gaps widen?

c. What are the most important ways organizations across the system have supported the [name of local HPOG program]?

d. Are there activities that organizations would like to implement to support the local system but have not been able to undertake yet? What are the challenges to doing so?

e. What are the main factors that hindered implementation of [name of local HPOG program]?

f. Anything else you would like to add that we haven’t discussed?

6. Identifying the most involved partner organizations (for HPOG program operators only)

As we mentioned earlier, we would like your help in identifying up to seven of your partner organizations that you work with that could also provide their perspectives on the service delivery system in which [name of local HPOG program] operates. We hope to interview representatives from different types of organizations such as education and training providers, workforce agencies, community- or faith-based organizations, employers and industry organizations, or others. We also want to interview both partners who have been highly involved in your HPOG program but also partners who have been less involved. Now that you have a sense of the questions we are asking, which organizations do you recommend we talk to and who at those organizations would be most appropriate for an interview? Let’s go through the list of partners we discussed earlier in the interview. Knowing that we are also interested in partnerships that did not go so well, please identify some potential partners to whom you reached out or considered reaching out, but little or nothing was accomplished.

After this interview, we are going to review our notes and select the partners we would like to interview for this study. In approximately a week, we will send a follow-up email asking for contact information for the partners we would like to interview. We appreciate your help in contacting these partners.

Thank you for your time today. We took a lot of notes today but we want to be sure our notes are accurate. If we have anything we need to clarify, is it okay if we call or email you?