#2.Program implementation staff interview

Initial Site Visit Protocol – Program Implementation Staff

INITIAL (YEAR 1) INTERVIEW PROTCOL

Tribal HPOG Program Implementation Staff - Interview

The purpose of the interview is to obtain information from Program Implementation Staff about their involvement in developing and implementing the tribal HPOG program. The following topics are addressed:

A. Background 2

B. Planning 3

C. Program Structures 3

1. Program Design and Curriculum 3

2. Contextual Factors 4

D. Program Processes 4

1. Recruitment 4

2. Orientation 4

3. Supportive Services 4

4. Incorporation of Family Education Model 5

5. Implementation Facilitators and Challenges 5

6. Quality of Instruction 5

7. Use of Program Data 6

E. Participant Outcomes 6

1. Participant Educational Attainment 6

2. Participant Employment Outcomes 6

3. Participant Employability Outcomes 7

4. Role of tribal HPOG program in building tribal health workforce capacity 7

5. Satisfaction with tribal HPOG program 7

F. Recommendations for Program Improvement 8

G. Conclusion 8

All background information relevant to these topics will be consulted prior to the interview in order to provide contextual information.

Tribal HPOG program	Blackfeet Community College Cankdeska Cikana Community College Cook Inlet Tribal Council College of Menominee Nation Turtle Mountain	Study ID
Interviewed by		Date & time

This interview is being conducted for the Evaluation of the Tribal Health Professions Opportunity Grants. The Health Profession Opportunity Grants (HPOG) program is administered by the Administration for Children and Families (ACF), an agency within the U.S. Department of Health and Human Services. ACF funded 32 five-year demonstration projects to design and implement innovative health workforce development training programs targeting Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. Five of the 32 demonstration projects were awarded to Tribal Organizations and Tribal Colleges to develop culturally-informed training programs. The Tribal HPOG Evaluation is a comprehensive evaluation of the design, implementation, and outcomes of the five Tribal HPOG programs. The interview questions will focus on your perceptions of the Tribal HPOG program, including the program design and curriculum, recruitment, supportive services, family engagement, the quality of instruction, educational attainment and employment outcomes of participants, implementation barriers and facilitators, and overall satisfaction. Your participation is voluntary, but it is very important because your responses will help us to improve the program. As explained in the consent form you signed, we will keep information about you private and you will not be identified in any report or publication of this study or its results. You may decline to answer any question you wish. If you have any questions, please let me know.

NOTE: This interview protocol will be tailored based on the specific role of an individual. All sections may not be applicable to every individual. It is unlikely that any one individual will be asked all questions in this protocol.

Background

1. What is the name of your agency or organization?

2. How long has it been in existence? What is its history?

3. What are the range of services and programs provided?

4. What is the agency’s or organization’s service area?

5. What is your role in the organization/agency?

6. What is your role and responsibilities for the Tribal HPOG project?

7. How were you recruited to work on the Tribal HPOG project?

8. Describe your professional background. Have you worked with Tribal populations before?

Planning

9. Were you involved in planning for the program? YES NO

If no, skip to the next section.

If yes, continue with the questions below.

10. Who else was involved in the planning process? How were different viewpoints incorporated into program planning?

11. What issues were addressed? Were there challenges that required a compromise?

Program Structures

Have grantees incorporated structures necessary to enhance the health care workforce needs of the community?

Program Design and Curriculum

12. [READ ALOUD TO RESPONDENTS BEFORE ASKING 12a-12f] We are not interested in learning more about the program curriculum, such as academic lectures, use of field practicum training manuals, and in learningabout ways in which the program was designed or modified for Tribal populations.

    1. Describe the career pathways provided by your program.

• Describe the licensure/certification process for each career pathway, as applicable.

• How portable is the licensure/certificate?

2. Describe the elements of the training program and its curriculum/curricula.[if detail made available in other program materials, skip specifics below]

• Competencies to be developed (e.g., skill requirements of the target occupation)

• Pre-requisites

• Duration of training program

• Who provides training

• Program location

  • Convenience

  • Accessibility

3. Why was this curriculum/model/approach chosen? Were other curricula/models/approaches considered? If so, which ones?

4. Were adaptations or modifications made to the training program based on local conditions or preferences?

5. Was the model/curriculum adapted to be culturally relevant? If yes, then what adaptation strategies are used to deliver a culturally relevant curriculum?

6. [If program instructor] Do you have previous experience teaching this curriculum to Tribal students? If not, did you do anything to prepare to work with this population? Please describe.

Contextual Factors

13. [READ ALOUD TO RESPONDENTS BEFORE ASKING 13a-13b] Now we would like to learn about how the social, economic, and political context of the community influence program design and implementation.

1. Describe any barriers to education/training and employment for population served. These might include Socio-cultural barriers include language and communication differences; practices that differ from their own beliefs and traditions; fear and mistrust of [TBD] institutions, and a lack of knowledge about how to navigate the system. Barriers include childcare, housing, transportation, health, mental health, substance abuse, domestic violence, GED, training, adult basic education, English language learning.

2. What community factors influence the design and implementation of the program (e.g., ongoing recession, organization priorities, social tension)? Please describe.

Program Processes

Have grantees implemented processes to prepare participants for employment in the tribal health care sector?

Recruitment

14. [READ ALOUD TO RESPONDENTS BEFORE ASKING 14a-14h] Now we would like to learn about how participants found out about the program, what recruitment strategies were utilized, and if those strategies were effective.

1. What are the enrollment requirements?

2. Describe participant demographics.

3. How were participants identified/targeted for the program?

4. What are the referral sources?

5. What recruitment methods were used? What methods were most effective?

6. How are participants screened before their suitability for program participation or services is determined?

7. Did you have more/fewer participants than anticipated?

   1. If more, was a wait list developed?

   2. If fewer, what did you do encourage interest?

8. Did you experience any challenges in recruiting participants? If so, describe.

Orientation

15. [READ ALOUD TO RESPONDENTS BEFORE ASKING 15a-15e] Building on the last questions, we would now like to discuss how participants were introduced and welcomed into the program.

1. How are students welcomed/oriented to the program?

2. Who conducted the orientation?

3. How is their training plan developed? Using what assessment instruments?

4. What orientation strategies facilitated program participation and completion? Describe.

5. Describe any challenges experienced in orientation and program participation. What strategies were used to overcome the challenge(s) noted?

Supportive Services

16. [READ ALOUD TO RESPONDENTS BEFORE ASKING QUESTIONS 16a-16j] Next we would like to ask you some questions about the support services that are offered as part of the program and how they are incorporated.

1. What assessments were conducted to determine needs (participant and/or family)? Are needs re-assessed over time and if so, how?

2. What types of services or incentives are being provided to participants?

• Social services (e.g., food stamps, childcare, transportation)

• Employability services (e.g., essential skills, life skills, job readiness)

• Employment related services (e.g., job development and placement, job coaching)

• Program retention services (e.g., mentoring)

• Job retention services (e.g., mentoring, peer support groups)

3. Who provides the service?

• The tribe (administration, departments, programs)?

• Tribal organizations?

• Non-tribal partners (IHS, local social services)

4. Are the supportive services provided on-site or off-site?

5. How do participants know about/are made aware of the supportive services that are available?

6. What are participants’ help-seeking behaviors?

7. Are supportive services culturally-based? If so, please describe.

8. How are service components sequenced and coordinated? How are they designed to address participants’ needs and unique barriers to employment?

9. Describe any challenges experienced in providing supportive services. What strategies were used to overcome the challenge(s) noted?

10. Which supportive services, if any, have been most effective at enabling students to participate in and complete the program?

Incorporation of Family Education Model

17. [READ ALOUD TO RESPONDENTS BEFORE ASKING 17a-17c] We would like to discuss any strategies that are used to engage the families of program participants.

    1. What strategies are used to engage families to support the participant?

• Participation in orientation

• Participation in college activities

• Provide support for studying

• Other

2. Describe any challenges experienced in engaging families. What strategies were used to overcome the challenge(s) noted?

3. What strategies facilitated program participation and completion? Describe.

Implementation Facilitators and Challenges

18. [READ ALOUD TO RESPONDENTS BEFORE ASKING 18a-18e] Next we would like to ask you some questions that will help us to understand whether or not the program was implemented as intended and what barriers you faced in implementing the program.

    1. Have you experienced any start-up challenges? If yes, please describe.

    2. Have you experience any implementation challenges? If yes, please describe.

    3. Did you receive any training for your position? If yes, describe. Did the training help you effectively implement the curriculum? What aspect(s) were most helpful?

    4. Are you supervised? If yes, has this supervision been helpful and consistent?

    5. Do you receive mentorship from program leadership and/or your peers?

Quality of Instruction

19. [READ ALOUD TO RESPONDENTS BEFORE ASKING 19a-19b] Next we would like to discuss the quality of instruction delivered through the program.

1. As measured by core competencies [list core competencies]?

2. As measured by student perceptions?

Use of Program Data

20. [READ TO RESPONDENT BEFORE ASKING 20a-20c] Now we would like to ask a few questions about the collection and use of program data.

    1. Is there a process in place for entering data into the Performance Management System?

       1. How are data collected from partners?

       2. Who is responsible for entering data the Performance Management System?

    2. Do you use program data for program management decisions, performance monitoring or program correction? If yes, describe how these data are used.

    3. Are there any challenges to collecting and using the program data?

• If yes: Please explain.

• Have these issues been resolved? Do you need any additional technical assistance?

Participant Outcomes

Is there evidence that participants in the program achieved successful employment and work force capacity building outcomes?

[Participant Educational Attainment – Role of Tribal HPOG program in building Tribal Health Workforce Capacity --- ONLY ask if there are program completers at the time of site visit]

Participant Educational Attainment

21. [READ TO RESPONDENT BEFORE ASKING 21a-21h] We would like to learn more about the educational attainment of program completers, and if participation in the program resulted in a degree/certification, and/or leads to the successful receipt of a license or certification issued through a state or industry entity.

1. How do you define program completion?

2. How many completed training? How many did not complete the training? Do you have a sense for why students dropped out the program?

3. In general, have you found that program completers move onto a professional or industry recognized certificate or licensure?

4. How many students advanced to another degree program?

5. What, if any, core competencies were achieved [list core competencies]?

6. Were there other indicators of success identified and achieved?

7. What program components (e.g. competency based curricula, supportive/cultural services, family education/engagement/employment and employability activities) are important to program participation and completion?

8. Have you found that any of the program components are not as useful in supporting program participation and completion as anticipated?

Participant Employment Outcomes

22. [READ ALOUD TO RESPONDENTS BEFORE ASKING 22a-22d] Now we would like to ask you some questions about the employment outcomes of participants.

1. [If the program has an internship or practicum component] How do participants transition from the internship/practicum to employment?

2. What are participants’ employment outcomes?

• Employed full-time

• Employed part-time

• Serving internship

• Unemployed

3. What types of jobs do participants have? Provide examples. Are participants supervised or mentored while on the job/practicum site?

4. With what employers? (Note: Need to note tribal hiring preferences & policies for employment (tribal member, spouse of tribal member, member of another tribe)

1. Tribe

   1. In area

   2. Out of area

2. Other tribe

3. Non-tribal

   1. IHS

   2. Urban Indian Clinics

   3. Local health care

   4. Out of area

Participant Employability Outcomes

23. [READ TO RESPONDENTS BEFORE ASKING 23a] Next, we are also interested in learning other ways the program has impacted participants.

    1. Did employment result in reduced use of income supports?

       1. TANF or General Assistance

       2. SNAP (food stamps), Commodities

Role of tribal HPOG program in building tribal health workforce capacity

24. [READ TO RESPONDENTS BEFORE ASKING 24a-24b] Building on the last section, we would like to ask a few questions about whether the program has been able to help fill vacancies in the local or regional tribal health workforce.

    1. Are participants serving their own community or another Tribal community? Provide examples.

    2. Did participants encounter any barriers in finding employment in a Tribal community? If so, what were these barriers?

Satisfaction with tribal HPOG program

25. [READ ALOUD TO RESPONDENTS BEFORE ASKING 25a-25c] Now we would like to hear your perception of the satisfaction level of program stakeholders.

    1. On a scale of 1 to 5, where 1 is ‘not satisfied at all’ and 5 is ‘very satisfied’, how satisfied are you with the program?

    2. To your knowledge, are

1. participants satisfied with the program?

   1. Are you measuring participant satisfaction?

2. partners satisfied with the program?

   1. Are you measuring partner satisfaction?

3. employers satisfied with the program?

   1. Are you measuring employer satisfaction?

3. In your opinion, what have been some of the key benefits for participants served through the tribal HPOG program?

Recommendations for Program Improvement

26. Is there anything that you would change about the program that could be helpful to future participants?

Conclusion

27. Is there anything you would like to add before concluding the interview?

Thank you very much for your time. It has been a pleasure to speak with you.

Follow Up Site Visit Protocol – Program Implementation Staff

FOLLOW UP 1 AND 2 INTERVIEW PROTCOL

Tribal HPOG Program Implementation Staff - Interview

The purpose of the interview is to obtain information from Program Implementation Staff about their involvement in developing and implementing the tribal HPOG program. The following topics are addressed:

A. Background 10

B. Program Structures 10

1. Program Design and Curriculum 10

2. Contextual Factors 10

C. Program Processes 11

1. Recruitment 11

2. Orientation 11

3. Supportive Services 11

4. Incorporation of Family Education Model 12

5. Implementation Facilitators and Challenges 12

6. Quality of Instruction 12

7. Use of Program Data 13

D. Participant Outcomes 13

1. Participant Educational Attainment 13

2. Participant Employment Outcomes 13

3. Participant Employability Outcomes 14

4. Role of tribal HPOG program in building tribal health workforce capacity 14

5. Satisfaction with tribal HPOG program 14

E. Recommendations for Program Improvement 14

F. Conclusion 15

All background information relevant to these topics will be consulted prior to the interview in order to provide contextual information.

Tribal HPOG program	Blackfeet Community College Cankdeska Cikana Community College Cook Inlet Tribal Council College of Menominee Nation Turtle Mountain	Study ID
Interviewed by		Date & time

This interview is being conducted for the Evaluation of the Tribal Health Professions Opportunity Grants. The Health Profession Opportunity Grants (HPOG) program is administered by the Administration for Children and Families (ACF), an agency within the U.S. Department of Health and Human Services. ACF funded 32 five-year demonstration projects to design and implement innovative health workforce development training programs targeting Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. Five of the 32 demonstration projects were awarded to Tribal Organizations and Tribal Colleges to develop culturally-informed training programs. The Tribal HPOG Evaluation is a comprehensive evaluation of the design, implementation, and outcomes of the five Tribal HPOG programs. The interview questions will focus on your perceptions of the Tribal HPOG program, including the program design and curriculum, recruitment, supportive services, family engagement, the quality of instruction, educational attainment and employment outcomes of participants, implementation barriers and facilitators, and overall satisfaction. Your participation is voluntary, but it is very important because your responses will help us to improve the program. As explained in the consent form you signed, we will keep information about you private and you will not be identified in any report or publication of this study or its results. You may decline to answer any question you wish. If you have any questions, please let me know.

NOTE: This interview protocol will be tailored based on the specific role of an individual. All sections may not be applicable to every individual. It is unlikely that any one individual will be asked all questions in this protocol.

Background [Only if not previously interviewed]

1. What is the name of your agency or organization?

2. How long has it been in existence? What is its history?

3. What are the range of services and programs provided?

4. What is the agency or organization’s service area?

5. What is your role in the organization? For the Tribal HPOG project?

6. How were you recruited to work on the Tribal HPOG project?

7. Describe your professional background. Have you worked with Tribal populations before?

8. Were you involved in the planning for the program? If yes, please describe.

Program Structures

Have grantees incorporated structures necessary to enhance the health care workforce needs of the community?

Program Design and Curriculum

9. [READ ALOUD TO RESPONDENTS BEFORE ASKING 9a-9d] We are not interested in learning more about the program curriculum, such as academic lectures and the use of field practicum training manuals, and in learning about ways in which the program was designed or modified for Tribal populations.

   1. Are you aware of any changes to the program curriculum since the last time we spoke (or beginning of the program if new respondent)? If yes, please describe.

   2. Are there aspects of the program curriculum that you believe are more effective than others for program participation and completion? Please describe.

   3. Have you encountered any problems/challenges with the program curriculum? If yes, please describe.

   4. [If new respondent] Do you have previous experience teaching this curriculum to Tribal students? If not, did you do anything to prepare? Please describe.

Contextual Factors

10. [READ ALOUD TO RESPONDENTS BEFORE ASKING 10a-10b] Now we would like to learn about how the social, economic, and political context of the community influence program design and implementation.

1. Describe barriers to education/training and employment for population served. For example, socio-cultural barriers include language and communication differences; practices that differ from their own beliefs and traditions; fear and mistrust of [TBD] institutions, and a lack of knowledge about how to navigate the system. Barriers include childcare, housing, transportation, health, mental health, substance abuse, domestic violence, GED, training, adult basic education, English language learning/literacy.

2. What community factors influence the implementation of the program (e.g, ongoing recession, organization priorities, social tension)? Please describe.

Program Processes

Have grantees implemented processes that successfully prepare participants for employment in the Tribal health care sector?

Recruitment

11. [READ ALOUD TO RESPONDENTS BEFORE ASKING 11a-11d] Now we would like to learn about how participants found out about the program, what recruitment strategies were utilized, and if those strategies were effective

1. Are you aware of any changes to enrollment requirements since the last time we spoke (or beginning of the program if new respondent)? If yes, describe.

2. Are you aware of any changes to recruitment methods since the last time we spoke (or beginning of the program if new respondent)? What methods do you think were most effective for program participation and completion?

3. Did you have more/fewer participants than anticipated?

1. If more, was a wait list developed?

2. If less, what did you do encourage interest?

4. Did you experience any challenge in recruiting participants? If so, describe.

Orientation

12. [READ ALOUD TO RESPONDENTS BEFORE ASKING 12a-12c] Building on the last questions, we would now like to discuss how participants were introduced and welcomed into the program. ?

1. Are you aware of any changes to orientation strategies used to engage patients since the last time we spoke (or beginning of the program if new respondent)? If yes, describe.

2. What orientation strategies facilitated program participation and completion? Describe.

3. Describe any challenges experienced in orientation and program participation. What strategies were used to overcome the challenge(s) noted?

Supportive Services

13. [READ ALOUD TO RESPONDENTS BEFORE ASKING QUESTIONS 13a-13e] Next we would like to ask you some questions about the support services that are offered as part of the program and how they are incorporated.

1. Are you aware of any changes to the types of supportive services being provided to participants since the last time we spoke (or beginning of the program if new respondent)? If yes, please describe.

   • Social services (e.g.., food stamps, childcare)

   • Employability services (e.g.., essential skills, life skills, job readiness)

   • Employment related services (e.g.., job development and placement, job coaching)

   • Program retention services (e.g., mentoring)

   • Job retention services (e.g., mentoring, peer support groups)

2. Have students utilized these services? Which services/group of services have been utilized most frequently? What are participants’ help-seeking behaviors?

3. What support services facilitated participant success? Describe.

4. Have students utilized any virtual services (e.g.., online tools for job-seekers)? If yes, have they been effective?

5. Describe any challenges experienced in providing supportive services. What strategies were used to overcome the challenge(s) noted?

Incorporation of Family Education Model

14. [if family engagement model part of the program] [READ ALOUD TO RESPONDENTS BEFORE ASKING 14a-14c] Next, we would like to discuss any strategies that are used to engage the families of program participants.

    1. Are you aware of any changes to strategies used to engage families to support the participants since the last time we spoke (or beginning of the program if new respondent)?

    2. Describe any challenges experienced in engaging families. What strategies were used to overcome the challenge(s) noted?

    3. What strategies, if any, have facilitated participant success? Describe.

Implementation Facilitators and Challenges

15. [READ ALOUD TO RESPONDENTS BEFORE ASKING 15a-15d] Next we would like to ask you some questions that will help us to understand whether or not the program was implemented as intended and what barriers you faced in implementing the program.

1. Have you experienced any implementation challenges? If yes, please describe.

2. [if new respondent] Did you receive any training for your position? If yes, describe. Did the training help you effectively implement the curriculum? What aspect(s) were most helpful?

3. Has there been any implementation staff turnover? If yes, how was this handled? Do you think this had an effect on program implementation?

4. Describe the supervision you have received since we last spoke (or since you started the program)? Do you receive mentorship from program leadership and/or your peers?

Quality of Instruction

16. [READ ALOUD TO RESPONDENTS BEFORE ASKING 16a-16b] Next we would like to discuss the quality of instruction delivered through the program.

1. As measured by core competencies [list core competencies]?

2. As measured by student perceptions?

Use of Program Data

17. [READ TO RESPONDENT BEFORE ASKING 17a-17b] Now we would like to ask a few questions about the collection and use of program data.

    1. Do you use program data for program management decisions, performance monitoring or program correction? If yes, describe how these data are used.

    2. Are there any challenges to collecting and using the program data?

• If yes: Please explain.

• Have these issues been resolved? Do you need any additional technical assistance?

Participant Outcomes

Is there evidence that participation in the program resulted in successful employment and work force capacity building outcomes?

Participant Educational Attainment

18. [READ TO RESPONDENT BEFORE ASKING 18a-18f] We would like to learn more about the educational attainment of program completers, and if participation in the program resulted in a degree/certification, and/or leads to the successful receipt of a license or certification issued through a state or industry entity.

1. How many completed training? How many did not complete the training? Do you have a sense for why students have dropped out the program?

2. In general, do program completers move onto a professional or industry recognized certificate or licensure?

3. How many students advanced to another degree program?

4. What, if any, competencies were achieved [list competencies]?

5. Were there other indicators of success identified and achieved?

6. What program components (e.g. competency based curricula, supportive/cultural services, family education/engagement/employment and employability activities) are important to program participation and completion?

Participant Employment Outcomes

19. [READ ALOUD TO RESPONDENTS BEFORE ASKING 19a-19h] Now we would like to ask questions about the employment outcomes of participants.

1. [If program has an internship or practicum component] How do participants transition to employment?

2. What are participants’ employment outcomes?

• Employed full-time

• Employed part-time

• Serving internship

• Unemployed

3. What types of jobs do participants have? Provide examples. Are participants supervised or mentored while on the job/practicum site?

4. With what employers? (Note: Need to note tribal hiring preferences & policies for employment (tribal member, spouse of tribal member, member of another tribe)

   1. Tribe

      1. In area

      2. Out of area

   2. Other tribe

   3. Non-tribal

      1. IHS

      2. Urban Indian Clinic

      3. Local health care

      4. Out of area

5. Are you aware of whether participants have retained their jobs? For how long? Were retention services provided?

6. Are you aware of how participant earnings compare to pre-participation earnings?

7. Do you know if any have experience some type of employment advancement? Position? Higher pay? More hours? Responsibilities?

Participant Employability Outcomes

20. [READ TO RESPONDENTS BEFORE ASKING 20a] We are also interested in learning other ways the program has impacted participants.

    1. Did employment result in reduced use of income supports?

       1. TANF or General Assistance

       2. SNAP (food stamps), Commodities

Role of tribal HPOG program in building tribal health workforce capacity

21. [READ TO RESPONDENTS BEFORE ASKING 21a-21b] Building on the last section, we would like to ask a few questions about whether the program has been able to help fill vacancies in the local or regional tribal health workforce.

1. Are participants serving their own community or another Tribal community? Provide examples. If not serving Tribal populations, why not?

2. Did participants encounter any barriers in finding employment in a Tribal community? If so, what were these barriers?

Satisfaction with tribal HPOG program

22. [READ ALOUD TO RESPONDENTS BEFORE ASKING 22a-22c] Now we would like to hear your perception of the satisfaction level of program stakeholders.

    1. On a scale of 1 to 5, where 1 is “not satisfied at all” and 5 is “very satisfied”, how satisfied are you with the program?

    2. To your knowledge, are

1. participants satisfied with the program?

   1. Are you measuring participant satisfaction?

2. partners satisfied with the program?

   1. Are you measuring partner satisfaction?

3. employers satisfied with the program?

   1. Are you measuring employer satisfaction?

3. In your opinion, what have been some of the key benefits for participants served through the tribal HPOG program?

Recommendations for Program Improvement

23. Is there anything that you would change about the program that could be helpful to future participants?

Conclusion

24. Is there anything you would like to add before concluding the interview?

Thank you very much for your time. It has been a pleasure to speak with you.

15