Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews

S upporting Statement for OMB Clearance Request

Instrument 6: HPOG 2.0 Tribal Evaluation Grantee and Partner Administrative Staff Interviews

National and Tribal Evaluation of the 2^nd Generation of the Health Profession Opportunity Grants (HPOG)

0970-0462

September 2016

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Nicole Constance, Hilary Forster, and Amelia Popham

Grantee and Partner Administrative Staff

Initial (Year 1) Interview Protocol

This interview is being conducted for the evaluation of the second round of the Tribal Health Profession Opportunity Grants (HPOG) Program. The HPOG Program is administered by the Administration for Children and Families (ACF), an agency within the U.S. Department of Health and Human Services. ACF funded 32 five-year demonstration projects to provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for healthcare occupations. Five of the 32 demonstration projects were awarded to Tribal Organizations and Tribal Colleges to develop culturally-informed training programs. The Tribal HPOG Evaluation is a comprehensive evaluation of the design, implementation, and outcomes of the five Tribal HPOG programs. The interview questions will focus on your perceptions of the Tribal HPOG program, including the program design and curriculum, partnerships, recruitment, supportive services, family engagement, the quality of instruction, educational attainment and employment outcomes of participants, implementation barriers and facilitators, and overall satisfaction. The interview will take about 60 minutes to complete. Your participation is voluntary, but it is very important because your responses will help us to improve the program. We will keep information about you private and you will not be identified in any report or publication of this study or its results. You may decline to answer any question you wish. If you have any questions, please let me know.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0462 and the expiration date is XX/XX/XXXX.

Note: This interview protocol will be tailored based on the specific role of an individual. All sections will not be applicable to every individual. No one individual will be asked all questions in this protocol.

Background

1. Please describe your agency or organization.

   1. What are the range of services and programs provided? To what degree was the grantee serving similar populations with similar programs prior to receiving HPOG funding?

2. What is your role in the organization/agency? How long have you been in this role? How long have you been with the organization?

3. What is your role and responsibilities for the [HPOG program]?

   1. Tell us how you became involved in the [HPOG program].

   2. Briefly describe your educational and professional background as it relates to your role on the [HPOG program]. Have you worked with Tribal populations before? If yes, describe.

Planning

4. [Partners only] How was your agency invited to participate in the [HPOG program] by [HPOG grantee]?

5. [HPOG Grantee Staff only] Tell me about your agency’s decision to apply for the Tribal HPOG Program. What did you expect or hope for (e.g., opportunity to work with partners, interest in health professions focus of the program)?

6. Were you involved in planning for the program? YES NO

If no, skip to the next section.

If yes, continue with the questions below.

7. What was your role in planning for the program?

8. How were different viewpoints incorporated into program planning? Probe: whose viewpoints were considered/incorporated, e.g., employers, partners, students, community members, etc.

9. What issues were addressed? Were there challenges that required a compromise? Explain.

Program Structures

Program Design

10. What is the administrative structure of the [HPOG program]?

    1. What organizations are responsible for each of the program components (e.g., academic instruction, recruitment, provision of supportive services, etc.)?

    2. What partnerships have been formed to deliver training or program services?

11. How is the [HPOG program] integrated into the overall structure of the [HPOG grantee]?

    1. Were there staffing changes made to implement the [HPOG program]?

    2. Are all positions filled? Were there any challenges in staffing the program?

12. How are program implementation and design decisions made?

    1. What community factors influence the design and implementation of the program (e.g., tribal or regional economic conditions or opportunities, economic development, effects of the recession, tribal or organizational priorities)? Please describe.

13. What, if any, resources have been leveraged to support this program?

14. Have any HPOG funds been used to make changes improvements to existing facilities or been used to purchase equipment for the [HPOG program] (e.g., lab equipment or supplies, equipment for distance learning)?

15. Is the program administered through multiple implementation sites? If so, are there differences in program design or implementation by site?

16. Did the [HPOG grantee] request a waiver for any required program elements per the funding opportunity announcement (e.g., apprenticeship, other)?

Partnerships

[Lead organization only]

17. What partnerships have been formed to deliver training or program services? Which partners are tribal vs. non-tribal? Specifically, what, if any, is the role of:

    1. Public human service agencies (e.g., TANF, housing, substance abuse, disability and other agencies)

    2. Public workforce investment system (e.g., Workforce Investment Act programs)

    3. Unions

    4. Employer and employment agencies

    5. Education and training institutions

    6. Faith-based and community-initiatives

    7. Other service providers?

18. Describe your partnerships with employers? Formal or informal?

19. Have you worked with any of the [HPOG program] partners before?

    1. Describe recruitment and outreach to project partners, including potential education institutions and employers.

20. What strategies are used for collaboration and coordination across all project partners? How are they established (formal and informal)?

    1. How is the partnership functioning? Are partners meeting their obligations (per the MOU, if applicable)?

    2. Have you run into any challenges? If yes, please describe.

    3. Have you had any successes? If yes, please describe.

21. Are community engagement strategies used to inform project planning and implementation? If yes, what kinds (e.g. advisory boards, council of elders, open community meetings/gatherings)?

[Partners only]

22. Describe your relationship with the [HPOG grantee].

    1. Have you worked with this organization before?

    2. How is the partnership functioning? Have you run into any challenges? If yes, please describe.

23. Aside from [HPOG grantee] do you work closely with any of the other organizations working with [HPOG Grantee] on the [HPOG program]? If yes:

    1. Have you worked with this organization before?

    2. How is the partnership functioning? Have you run into any challenges? If yes, please describe.

    3. Have you had any successes? If yes, please describe.

Curriculum

24. What academic or training programs are offered by [HPOG grantee]? What is the program curriculum (e.g. academic lectures, field practicum training manual)?

25. How did [HPOG grantee] decide to offer trainings for these occupations (e.g., based on labor market information and local demand)?

    1. Were tribal and/or local (i.e., off –reservation. surrounding area) workforce needs assessed? If so, how?

    2. Were other sources of information/data were used to determine the fit between the training program and local industry needs? If so, what?

26. We understand your instructional models are [types of instructional models being used]. How did you decide to use this model(s)? What is the evidence base for the strategy or model? Are there any other instructional models or strategies used? In your opinion, which seem to be associated with positive outcomes?

    1. PROBE: accelerated learning approaches: Modularized learning; I-BEST; Contextualized learning; Team teaching; Self-paced learning; Prior learning assessments; other?

    2. PROBE: Simulations; Assessment; Distance learning/on-line learning; Real time on-line instruction; other?

27. Was the strategy or model adapted? Was the strategy or model adapted in any way to be culturally relevant? If yes, please describe.

28. Describe the elements of the training program(s) and its curriculum/curricula. [Repeat for each program. If detail made available in other program materials, skip specifics below.]

1. Competencies to be developed (e.g., skill requirements of the target occupation)

2. Pre-requisites

3. Duration of training program

4. Who provides training

5. Program location

   1. Convenience

   2. Accessibility

29. Why was this curriculum/approach chosen? Was it adapted to fit your community or to make it more culturally-relevant? If yes, please describe.

30. Were adaptations or modifications made to the training program(s) based on local conditions or preferences? Based on partner input? If yes, please describe.

31. Does [HPOG program] expose students to information about workplace culture, including industry attendance policies and on-the-job behavioral norms? If yes, how so?

32. What is your impression of the quality of instruction?

Implementation Staff Qualifications, Orientation and Retention

33. How were staff responsible for implementing the program curriculum recruited and/or selected?

    1. What are the qualifications that you considered?

    2. Were you able to hire staff that met these qualifications?

    3. Does staff have experience working with tribal populations?

    4. Did you face any challenges in finding staff with the qualifications that you are looking for?

34. Describe the training for staff responsible for implementing the program curriculum. How were they oriented to the program?

35. Describe the process for supervision and communication with program implementation staff. Are these staff mentored?

36. Has there been any implementation staff turnover? If yes, how was this handled? Do you think this had an effect on program implementation?

Contextual Factors

37. Does [HPOG grantee], tribe, or local community have any economic development activities related to healthcare? Please describe.

38. What community characteristics shape participants’ employment opportunities? (To include general labor market conditions in the area, the extent and nature of job opportunities, and industry skill initiatives).

39. Describe any challenges or barriers to education/training and employment for population served. These might include:

    1. Socio-cultural barriers, including language and communication differences; practices that differ from their own beliefs and traditions; fear and mistrust of institutions, and a lack of knowledge about how to navigate the system; or,

    2. Other economic or social barriers, including childcare, housing, transportation, health, mental health, substance abuse, domestic violence, GED, training, adult basic education, English language learning.

Use of Program Data

[Lead organization only]

40. How are data entered in PAGES? Who is responsible for keeping participant records current?

41. Do other administrative data systems support HPOG programming? If so, what kinds? Who operates those systems and for what purposes?

42. Do organizations/staff share information about program participants? What protocols are in place to share this information?

43. Do you share these data with partners? Stakeholders? Participants?

44. Are data used for program management decisions, performance monitoring, or program correction? If so, please explain.

45. Are there any challenges to collecting and using the program data? If so, please explain.

46. Have you had successes collecting and using the program data? If so, please explain.

Program Processes

Supportive Services

47. What supportive services are offered by the [HPOG program], partner organizations, or others and how are they incorporated?

    1. Academic services (e.g., mentoring, tutoring)

    2. Social services (e.g., food assistance, childcare, transportation)

    3. Employment related services (e.g., job development and placement, resume development, mock interviews, soft skills training, career transitional)

48. Who provides the supportive services (e.g., the grantee, a partner organization, etc.)?

    1. What internal partnerships or relationships have been formed to implement the supportive service?

    2. What external partnerships have been formed to provide the supportive services?

49. For each new type of supportive service provided (academic, social, employment-related), have any resources been leveraged?

50. What assessments were conducted to determine needs (of participant and/or family)? Are needs re-assessed over time and if so, how and how often?

51. How is participant eligibility for supportive services determined? If multiple implementation sites, are the eligibility criteria for supportive services the same across implementation sites?

52. How do participants know about/are participants made aware of the supportive services that are available?

53. Are any supportive services culturally-based? If so, please describe.

54. How are supportive services sequenced and coordinated? Are they designed to address participants’ needs and unique barriers to employment? If yes, how so?

55. For any of the supportive services described, including [list of supportive services offered by grantee], are they provided to participants during the training period only? Does the intensity of support decrease with the length of enrollment? Are “bridge programs” (i.e., post-training or intervention supportive services) provided when a student completes the course of study to facilitate his/her transition to further education or employment?

56. Describe any challenges experienced in providing supportive services. What strategies are used to overcome the challenge(s) noted?

57. Describe any successes experienced in providing supportive services.

58. In your opinion, which supportive services, if any, have been most effective at enabling students to participate in and complete the program?

59. Does your organization provide supportive services to help parents and their children succeed? Or elders? Or extended family?

    1. What types of family supportive services are provided to HPOG participants? Does the [HPOG program] partner with any child-focused programs?

PROBE: Child/dependent care; transportation; flexible scheduling; tuition reimbursement; other? Is the program tracking any child-related outcomes?

Implementation Facilitators and Challenges

60. What kinds of things helped with implementing of the [HPOG program]?

61. Is the [HPOG program] being implemented as intended (e.g., the proposed number of training sessions are being delivered)?

62. Have you experienced any start-up challenges? If yes, please describe.

63. What have you found to be helpful when starting-up the program?

64. Have you experienced any implementation challenges? If yes, please describe.

65. What have you found to be helpful when implementing the program?

66. [For returning HPOG 1.0 grantees only] How did your experience in HPOG 1.0 influence your implementation of HPOG 2.0?

    1. In what ways, if any, is your agency implementing HPOG 2.0 differently than HPOG 1.0? Why?

    2. Are any of these experiences different than what you experienced in HPOG 1.0? Why do you think that is?

Program Outcomes

Educational Attainment

67. As of [date], [insert number] students have completed training and [insert number] students did not complete training. In general, do you think students are progressing well in completing trainings?

68. Do you have a sense for why some students dropped out the program?

69. Have you found that generally participants who complete a training obtain a professional or industry recognized certificate or licensure?

70. Have you found that generally students complete a second training or advanced to another level of training or degree program? If students enroll in a second training, is there a pattern to the way most students move from one training to the next (e.g., are they employed while completing the second training, do they enter the workforce and then return to the program for continued training, do they go straight from a first training to a second training without working in between, etc.?)

71. Did the [HPOG program] recognize the accomplishments of participants? If yes, how so (e.g., graduation)? Are there other ways in which you recognized or celebrated success? Did you involve participants’ families in these activities?

Employment

72. In your opinion, how effective has the [HPOG program] been in helping participants find employment?

73. After completing training, are participants generally employed:

    1. Full time

    2. Part time

    3. Internship (i.e., unpaid position)

    4. Unemployed

74. Are participants generally finding employment in health professions or other industries? What types of jobs do participants have after completing training? Provide examples. Are any participants supervised or mentored by employers while on the job/practicum site?

75. Who are the major employers that have hired program participants? Where are these employers located (i.e., on reservation, off reservation, surrounding area, or not in the surrounding area)?

    1. Health professions? (Healthcare center, pharmacy, etc.?)

    2. Tribal employers?

    3. Non-tribal employers?

76. Do any of the employers have tribal hiring preferences and polices?

77. In your opinion do you think there are challenges that program participants will face in finding employment in a tribal community or other community? If so, what would these challenges be?

78. Do you have a process in place to track whether or not participants are receiving TANF or other benefits after they complete the program? Can you describe this process? Do you have any information about how many participants are no longer receiving services? Or still receiving services?

    1. If yes, can you provide us with this information?

79. Have you observed any changes in students’ goal setting and/or optimism about their future after completing the program? Can you share any examples?

80. What other outcomes or examples of participant success do you think participants achieved? PROBE: increased life skills, self-efficacy, confidence, proactive decision-making, reduced use of income supports, role modeling for their children or others, etc.

Health Professions Workforce Capacity Building

81. Of the following topics, has the [HPOG program] helped build capacity in any of the following areas?

    1. Developing sustainable curricula and learning modalities?

    2. Increasing instructional capacity?

    3. Obtaining equipment or facility upgrades?

    4. Expanding a network of supportive service providers or a referral system?

    5. Changing or expanding transfer or articulation agreements?

    6. Developing community college or workforce organization partnerships?

    7. Developing employer/industry partnerships or alliances?

    8. Creating a strategic alignment with public workforce investment system?

    9. Generating revenue?

    10. Growing or replicating programs?

    11. Are there any other areas that we haven’t included on this list?

Program Satisfaction

[Lead organization only]

82. In your opinion, are stakeholders satisfied with the [HPOG program]? Please explain.

    1. Administrative staff and program implementation staff?

    2. Partners?

    3. Employers?

    4. Participants?

    5. Tribal leadership and community members?

[Partner organization only]

83. Are you satisfied with the [HPOG program]? Why or why not?

84. Do you think other stakeholders are satisfied with the program? Please explain.

    1. Administrative staff and program implementation staff?

    2. Other Partners?

    3. Employers?

    4. Participants?

    5. Tribal leadership and community members?

[Lead and partner organizations]

85. In your opinion, what have been some of the key benefits for participants served through the [HPOG program]? For employers? Partners? For your organization?

86. Is there anything that you would change about the program that could be helpful to future participants? Employers? Partners? For your organization?

Conclusion

87. Is there anything you would like to add before concluding the interview?

Thank you for your participation.

Grantee and Partner Administrative Staff

Follow up Interview Protocol

This interview is being conducted for the evaluation of the second round of the Tribal Health Professions Opportunity Grants (HPOG) Program. The HPOG Program is administered by the Administration for Children and Families (ACF), an agency within the U.S. Department of Health and Human Services. ACF funded 32 five-year demonstration projects to provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for healthcare occupations. Five of the 32 demonstration projects were awarded to Tribal Organizations and Tribal Colleges to develop culturally-informed training programs. The Tribal HPOG Evaluation is a comprehensive evaluation of the design, implementation, and outcomes of the five Tribal HPOG programs. The interview questions will focus on your perceptions of the Tribal HPOG program, including the program design and curriculum, partnerships, recruitment, supportive services, family engagement, the quality of instruction, educational attainment and employment outcomes of participants, implementation barriers and facilitators, and overall satisfaction. The interview will take about 60 minutes to complete. Your participation is voluntary, but it is very important because your responses will help us to improve the program. We will keep information about you private and you will not be identified in any report or publication of this study or its results. You may decline to answer any question you wish. If you have any questions, please let me know.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0462 and the expiration date is XX/XX/XXXX.

Note: This interview protocol will be tailored based on the specific role of an individual. All sections will not be applicable to every individual. No one individual will be asked all questions in this protocol.

Background

1. Please describe your agency or organization.

   1. What are the range of services and programs provided? To what degree was the grantee serving similar populations with similar programs prior to receiving HPOG funding?

2. What is your role in the organization/agency? How long have you been in this role? How long have you been with the organization?

3. What is your role and responsibilities for the [HPOG program]?

   1. Tell us how you became involved in the [HPOG program].

   2. Briefly describe your educational and professional background as it relates to your role on the [HPOG program]. Have you worked with Tribal populations before? If yes, describe.

Program Structures

Program Design

4. Are you aware of any changes to the program structure or partnerships since we last spoke?

5. How is the [HPOG program] integrated into the overall structure of the [HPOG grantee]?

   1. Are all positions filled? Were there any challenges in staffing the program?

6. How are program implementation and design decisions made?

   1. What community factors influence the design and implementation of the program (e.g., tribal or regional economic conditions or opportunities, economic development, effects of the recession, tribal or organizational priorities)? Please describe.

7. What, if any, resources have been leveraged to support this program?

8. Have any HPOG funds been used to make changes improvements to existing facilities or been used to purchase equipment for the [HPOG program] (e.g., lab equipment or supplies, equipment for distance learning)?

9. Is the program administered through multiple implementation sites? If so, are there differences in program design or implementation by site?

Partnerships

[Lead organization only]

10. Are you aware of any changes to the partnerships have been formed to deliver training or program services? Which partners are tribal vs. non-tribal? Specifically, what, if any, is the role of:

    1. Public human service agencies (e.g., TANF, housing, substance abuse, disability and other agencies)

    2. Public workforce investment system (e.g., Workforce Investment Act programs)

    3. Unions

    4. Employer and employment agencies

    5. Education and training institutions

    6. Faith-based and community-initiatives

    7. Other service providers?

11. Describe your partnerships with employers? Formal or informal?

12. What strategies are used for collaboration and coordination across all project partners? How are they established (formal and informal)?

    1. How is the partnership functioning? Are partners meeting their obligations (per the MOU, if applicable)?

    2. Have you run into any challenges? If yes, please describe.

    3. Have you had any successes? If yes, please describe.

13. Are community engagement strategies used to inform project planning and implementation? If yes, what kinds (e.g. advisory boards, council of elders, open community meetings/gatherings)?

[Partners only]

14. Describe your relationship with the [HPOG grantee].

    1. Have you worked with this organization before?

    2. How is the partnership functioning? Have you run into any challenges? If yes, please describe.

15. Aside from [HPOG grantee] do you work closely with any of the other organizations working with [HPOG Grantee] on the [HPOG program]? If yes:

    1. Have you worked with this organization before?

    2. How is the partnership functioning? Have you run into any challenges? If yes, please describe.

    3. Have you had any successes? If yes, please describe.

Curriculum

16. Are you aware of any changes to the program curriculum since the last time we spoke (or beginning of the program if new respondent)? If yes, please describe.

17. Are you aware of any changes to the instructional models being used since the last time we spoke? What is the evidence base for the strategy or model? Are there any other instructional models or strategies used? In your opinion, which seem to be associated with positive outcomes?

    1. PROBE: accelerated learning approaches: Modularized learning; I-BEST; Contextualized learning; Team teaching; Self-paced learning; Prior learning assessments; other?

    2. PROBE: Simulations; Assessment; Distance learning/on-line learning; Real time on-line instruction; other?

18. Was the strategy or model adapted? Was the strategy or model adapted in any way to be culturally relevant? If yes, please describe.

19. Describe the elements of the training program(s) and its curriculum/curricula. [Repeat for each program. If detail made available in other program materials, skip specifics below.]

1. Competencies to be developed (e.g., skill requirements of the target occupation)

2. Pre-requisites

3. Duration of training program

4. Who provides training

5. Program location

   1. Convenience

   2. Accessibility

20. Why was this curriculum/approach chosen? Was it adapted to fit your community or to make it more culturally-relevant? If yes, please describe.

21. Were adaptations or modifications made to the training program(s) based on local conditions or preferences? Based on partner input? If yes, please describe.

22. Does the [HPOG program] expose students to information about workplace culture, including industry attendance policies and on-the-job behavioral norms? If yes, how so?

23. What is your impression of the quality of instruction?

Implementation Staff Qualifications, Orientation and Retention

24. Are you aware of any changes to the staff responsible for implementing [HPOG program]? Did you face any challenges in finding staff with the qualifications that you are looking for?

25. Describe the training for staff responsible for implementing the program curriculum. How were they oriented to the program?

26. Describe the process for supervision and communication with program implementation staff. Are these staff mentored?

27. Has there been any implementation staff turnover? If yes, how was this handled? Do you think this had an effect on program implementation?

Contextual Factors

28. Does the [HPOG grantee], tribe, or local community have any economic development activities related to healthcare? Please describe.

29. What community characteristics shape participants’ employment opportunities? (To include general labor market conditions in the area, the extent and nature of job opportunities, and industry skill initiatives).

30. Describe any challenges or barriers to education/training and employment for population served. These might include:

    1. Socio-cultural barriers, including language and communication differences; practices that differ from their own beliefs and traditions; fear and mistrust of institutions, and a lack of knowledge about how to navigate the system; or,

    2. Other economic or social barriers, including childcare, housing, transportation, health, mental health, substance abuse, domestic violence, GED, training, adult basic education, English language learning.

Use of Program Data

[Lead organization only]

31. How are data entered in PAGES? Who is responsible for keeping participant records current?

32. Do other administrative data systems support HPOG programming? If so, what kinds? Who operates those systems and for what purposes?

33. Do organizations/staff share information about program participants? What protocols are in place to share this information?

34. Do you share these data with partners? Stakeholders? Participants?

35. Are data used for program management decisions, performance monitoring, or program correction? If so, please explain.

36. Are there any challenges to collecting and using the program data? If so, please explain.

37. Have you had successes collecting and using the program data? If so, please explain.

Program Processes

Supportive Services

38. Are you aware of any changes to the types of supportive services being provided to participants since the last time we spoke (or beginning of the program if new respondent)? If yes, please describe.

    1. Academic services (e.g., mentoring, tutoring)

    2. Social services (e.g., food assistance, childcare, transportation)

    3. Employment related services (e.g., job development and placement, resume development, mock interviews, soft skills training, career transitional)

39. Who provides the supportive services (e.g., the grantee, a partner organization, etc.)?

40. What assessments were conducted to determine needs (of participant and/or family)? Are needs re-assessed over time and if so, how and how often?

41. How is participant eligibility for supportive services determined? If multiple implementation sites, are the eligibility criteria for supportive services the same across implementation sites?

42. For any of the supportive services described, including [list of supportive services offered by grantee], are they provided to participants during the training period only? Does the intensity of support decrease with the length of enrollment? Are “bridge programs” (i.e., post-training or intervention supportive services) provided when a student completes the course of study to facilitate his/her transition to further education or employment?

43. Describe any challenges experienced in providing supportive services. What strategies are used to overcome the challenge(s) noted?

44. Describe any successes experienced in providing supportive services.

45. In your opinion, which supportive services, if any, have been most effective at enabling students to participate in and complete the program?

46. Does your organization provide supportive services to help parents and their children succeed? Or elders? Or extended family?

    1. What types of family supports are provided to HPOG participants? Does the [HPOG program] partner with any child-focused programs?

PROBE: Child/dependent care; transportation; flexible scheduling; tuition reimbursement; other? Is the program tracking any child-related outcomes?

Implementation Facilitators and Challenges

47. Is the program being implemented as intended (e.g., the proposed number of training sessions are being delivered)?

48. Have you experienced any implementation challenges? If yes, please describe.

49. What have you found to be helpful when implementing the program?

50. [For returning HPOG 1.0 grantees only] How did your experience in HPOG 1.0 influence your implementation of HPOG 2.0?

    1. In what ways, if any, is your agency implementing HPOG 2.0 differently than HPOG 1.0? Why?

    2. Are any of these experiences different than what you experienced in HPOG 1.0? Why do you think that is?

Program Outcomes

Educational Attainment

51. As of [date], [insert number] students have completed training and [insert number] students did not complete training. In general, do you think students are progressing well in completing trainings?

52. Do you have a sense for why some students dropped out the program?

53. Have you found that generally participants who complete a training obtain a professional or industry recognized certificate or licensure?

54. Have you found that generally students complete a second training or advanced to another level of training or degree program? If students enroll in a second training, is there a pattern to the way most students move from one training to the next (e.g., are they employed while completing the second training, do they enter the workforce and then return to the program for continued training, do they go straight from a first training to a second training without working in between, etc.?)

55. Did the HPOG program recognize the accomplishments of participants? If yes, how so (e.g., graduation)? Are there other ways in which you recognized or celebrated success? Did you involve participants’ families in these activities?

Employment

56. In your opinion, how effective has the [HPOG program] been in helping participants find employment?

57. After completing training, are participants generally employed

    1. Full time

    2. Part time

    3. Internship (i.e., unpaid position)

    4. Unemployed

58. Are participants generally employed in health professions or other industries? What types of jobs do participants have after completing training? Provide examples. Are any participants supervised or mentored by employers while on the job/practicum site?

59. Who are the major employers that have hired HPOG participants? Where are these employers located (i.e., on reservation, off reservation, surrounding area, or not in the surrounding area)?

    1. Health professions? (Healthcare center, pharmacy, etc.?)

    2. Tribal employers?

    3. Non-tribal employers?

60. Do any of the employers have tribal hiring preferences and polices?

61. In your opinion do you think there are challenges that HPOG participants will face in finding employment in a tribal community or other community? If so, what would these challenges be?

62. Do you have a process in place to track whether or not participants are receiving TANF or other benefits after they complete the program? Can you describe this process? Do you have any information about how many participants are no longer receiving services? Or still receiving services?

    1. If yes, can you provide us with this information?

63. Have you observed any changes in students’ goal setting and/or optimism about the future after completing the program? Can you share any examples?

64. What other outcomes or examples of participant success do you think participants achieved? PROBE: increased life skills, self-efficacy, confidence, proactive decision-making, reduced use of income supports, role modeling for their children or others, etc.

Health Professions Workforce Capacity Building

65. Of the following topics, has the [HPOG program] helped build capacity in any of the following areas?

    1. Developing sustainable curricula and learning modalities?

    2. Increasing instructional capacity?

    3. Obtaining equipment or facility upgrades?

    4. Expanding a network of supportive service providers or a referral system?

    5. Changing or expanding transfer or articulation agreements?

    6. Developing community college or workforce organization partnerships?

    7. Developing employer/industry partnerships or alliances?

    8. Creating a strategic alignment with public workforce investment system?

    9. Generating revenue?

    10. Growing or replicating programs?

    11. Are there any other areas that we haven’t included on this list?

66. Do you think the [HPOG program] or something similar will continue beyond the grant period if [HPOG grantee] does not continue to receive HPOG funding? Explain.

Program Satisfaction

[Lead organization only]

67. Do you think stakeholders are satisfied with the program? Please explain.

1. Administrative staff and program implementation staff?

2. Partners?

3. Employers?

4. Participants?

5. Tribal leadership and community members?

[Partner organization only]

68. Are you satisfied with the [HPOG program]? Why or why not?

69. Do you think other stakeholders are satisfied with the program? Please explain.

6. Administrative staff and program implementation staff?

7. Other Partners?

8. Employers?

9. Participants?

10. Tribal leadership and community members?

[Lead and partner organizations]

70. In your opinion, what have been some of the key benefits for participants served through the [HPOG program]? For employers? Partners? For your organization?

71. Is there anything that you would change about the program that could be helpful to future participants? Employers? Partners? For your organization?

Conclusion

72. Is there anything you would like to add before concluding the interview?

Thank you for your participation.