1 Opioid Recovery Coach Model Bundled Evaluation Instrumen

Submitted to:
AmeriCorps
250 E. St., SW

Washington, DC 20024

Submitted by:
ICF Incorporated, LLC

9300 Lee Highway

Fairfax, VA 22031
https://www.icf.com

May 3, 2021

Supporting Statement for Paperwork
Reduction Act Submissions

Peer Recovery Coach Model Bundled Evaluation

Instruments

CONTENTS

Grantee Staff Survey Protocol 2

Peer Recovery Coach Survey Protocol 9

Beneficiary (Comparison) Survey Protocol 19

Beneficiary Interview Protocol 25

Evaluation TA Pre–Post Outcome Surveys 27

Evaluation TA Post-Presentation Surveys 32

GRANTEE STAFF SURVEY PROTOCOL

Management of peer recovery coaches

1. What is your current role at [organization name]?

• Program assistant

• Program manager

• Director

• Other (please specify): ____________

2. Has your organization received funding or resources through the Federal Opioid Development grant? If yes, please specify: ____________

3. How many peer recovery coaches does your program employ? ____________

4. How many coaches are paid? ____________

5. How many coaches are part-time?

6. How many coaches are in recovery themselves?

7. Do you keep track of recovery coaches’ relapses?

• Yes

• No

8. Does your program have a monitoring and oversight plan?

• Yes

• No (If no, skip to question 11.)

9. How many individuals with an opioid use disorder does your organization currently serve? ____________

10. How many individuals with an opioid use disorder does your organization currently serve through peer recovery coaching? ____________

11. Does your organization have a process to maintain contact with clients after they enter the program?

• Yes

• No

12. How does your organization recruit peer recovery coaches?

• Internal recovery program/program graduates

• Community recovery programs

• Other community organizations

• Schools and universities

• Other (please specify): ____________

13. How important is it for a potential peer recovery coach to possess lived experience?

• Very important

• Important

• Neither important nor unimportant

• Unimportant

• Very unimportant

14. Our organization requires that peer recovery coaches are certified by the state of [State name].

• Yes

• No

• Other (please specify): ____________

15. Per week, how many hours are peer recovery coaches required to provide services? Please specify: ____________

16. How important is it that AmeriCorps allows recovery coaches to have scheduling flexibility and serve [part-time or near full-time] so that they can attend to their own recovery needs?

• Very important

• Important

• Neither important nor unimportant

• Unimportant

• Very unimportant

17. How important is it to your program that a volunteer does not relapse, in order to remain as a recovery coach?

• Very important

• Important

• Neither important nor unimportant

• Unimportant

• Very unimportant

Peer recovery coach training

18. Does your program have an onboarding process for peer recovery coaches, including supervision policies and required training?

• Yes

• No

• Other (please specify): ____________

19. Are peer recovery coaches at your organization required to complete a state or national certification before beginning employment?

• Yes (please specify what state/national credentialing agency your organization uses):
  ____________

• No

20. Are you using any training curriculum for peer recovery coaches?

• Yes (please specify): ____________

• No

21. What type of training curriculum are you using for peer recovery coaches at your organization?

• National training curriculum

• State training curriculum

• Organizational training curriculum

• We do not use a training curriculum

22. In which of the following areas does the program at [org name] offer employees or clients a culturally responsive treatment environment? (Select all that apply.)

• Organizational values

• Governance

• Client treatment planning

• Evaluation and monitoring

• Language service

• Workforce and staff development

• Organizational infrastructure

• Other (please specify):________

23. The program at [organization name] assesses and incorporates the following components into a client’s treatment. (Select all that apply.)

• English, bilingual, or multilingual fluency

• Racial, ethnic, and cultural identities

• Family and extended family concerns (including nonblood kinships)

• Trauma history

• Relationship and dating concerns

• Sexual and gender orientation

• Health concerns

• Beliefs about substance use, abuse, and dependence

• Beliefs about substance abuse treatment

• Family views on substance use and substance abuse treatment

• Treatment concerns related to cultural differences

• Cultural approaches to healing or treatment of substance use and mental disorders

• Work history and concerns

• Socio-economic and financial concerns

• Current network of support

• Community concerns

• Other (please specify): ____________

24. How is the training for peer recovery coaches offered? (Select all that apply.)

• One-on-one

• Group

• Online

• In person

25. How many hours of training are peer recovery coaches required to attend?

• Less than 1 hour

• 1-4 hours

• 5-8 hours

• 9-16 hours

• 17+ hours

26. Does your organization provide opportunities for peer recovery coaches to connect with each other?

• Yes

• No

Peer recovery coach services

27. How often do peer recovery coaches interact with individuals diagnosed with opioid use disorder?

• Daily

• One day a week or more

• One or two days a month

• A couple of times per year

• Less than two times per year

28. What other supports or services do you connect clients to? (Select all that apply.)

• Food assistance

• Transportation assistance

• Emergency shelter

• Physical or behavioral health providers

• Other (please specify): ____________

29. Please specify what community organizations your program works with to provide additional resources for clients with opioid use disorders and what services these organizations provide: ____________

• We do not work with community organizations

30. My program has the organizational capacity needed to provide services. (Using the selections below, please rate how strongly you agree with this statement.)

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

31. My program is able to leverage grant (i.e., financial) support. (Using the selections below, please rate how strongly you agree with this statement.)

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

32. My program is able to collaborate with partners, organizations, and community resources. (Using the selections below, please rate how strongly you agree with this statement.)

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

33. My program has received the support needed from AmeriCorps. (Using the selections below, please rate how strongly you agree with this statement.)

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

34. The criminal history background check is problematic. (Using the selections below, please rate how strongly you agree with this statement.)

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

35. The volunteer stipend is sufficient. (Using the selections below, please rate how strongly you agree with this statement.)

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree

36. Do you have any other comments? ____________

PEER RECOVERY COACH SURVEY PROTOCOL

Background

1. How long have you been with [grantee name] as volunteer?

• Less than 1 year

• 1-5 years

• 6-10 years

• 11-15 years

• 16+ years

2. How long have you worked as a volunteer?

• Less than 1 year

• 1-5 years

• 6-10 years

• 11-15 years

• 16+ years

3. What is your work status at [grantee name]?

• Full-time

• Part-time

4. Why did you choose to become a peer recovery coach? (Select all that apply.)

• Desire to help people seeking recovery from a substance use disorder

• As an alternative to full-time paid employment

• My own lived experience or recovery motivated me

• To help my community

• To learn new skills

• I enjoy working with people

• Other (please specify): ____________

5. I am certified by the state of [State name] in peer recovery coaching.

• Yes

• No

• Other (please specify): ____________

6. I have made/I am interested in making/I plan to make a career out of peer recovery coaching.

• Yes

• No

• Don’t know

Training

7. How many hours of peer recovery coach training have you received?

• Less than 1 hour

• 1-4 hours

• 5-8 hours

• 9-16 hours

• 17+ hours

8. Was the training you received in a group or one-on-one? (Select all that apply.)

• One-on-one

• Group

• Online

• In person

9. Did someone at your organization deliver the training, or was it delivered by someone outside of the organization?

• The instructor was from the organization

• The instructor was from outside the organization

• Don’t know

10. Did the training use a specific curriculum or manual?

• Yes (please specify): ____________

• No

• Don’t know

11. Did you find the training helpful?

• Yes

• No

Services

12. On a weekly basis, how many clients do you work with as a peer recovery coach?

• Less than 5 clients

• 5-10 clients

• 11-20 clients

• 21-30 clients

• 31+ clients

13. About how many hours per week do you work as peer recovery coach?

• Less than 1 hour

• 1-4 hours

• 5-8 hours

• 9-16 hours

• 17+ hours

14. Do you work with the same clients each week, or does your organization rotate clients among coaches?

• I work with the same clients each week.

• My organization rotates clients among coaches.

• Other (please specify): ____________

15. How often do you see each of the clients you work with?

• Daily

• One day a week or more

• One or two days a month

• A couple of times per year

• Less than two times per year

16. On average, how much time do you spend with each client per week?

• Less than 1 hour

• 1-4 hours

• 5-8 hours

• 9-16 hours

• 17+ hours

17. What mode of interactions do you have with clients? (Select all that apply.)

• In-person meetings

• Check-in calls

• Video calls (Zoom, Skype, FaceTime, etc.)

• Text messages

• Emails

• Other (please specify): ____________

18. Do you work with other coaches or medical personnel to help address the needs of clients with opioid use disorders?

• Yes

• No

• Don’t know

19. What services do you provide or facilitate as a peer recovery coach?

• Individual sessions/case management

• Group sessions

• Service referrals

• Other (please specify): ____________

20. The treatment plans I develop with my clients reflect their culture and worldviews. (Select the response that best fits your answer.)

• Completely

• Very well

• Somewhat

• Not very well

• Not at all

21. What other supports do you connect clients to?

• Emergency shelters

• Food services

• Employment services

• Physical or behavioral health providers

• Other services (please specify):

• I do not connect individuals to outside resources

Outcomes

22. How much do you agree or disagree that you get these benefits out of being a peer recovery coach? (Mark the box that best fits your answer for each benefit.)

    Benefits	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
    Satisfaction from helping individuals with opioid use disorders enter long-term recovery
    Satisfaction from improving the health of my community

23. How satisfied are you with the following aspects of being a peer recovery coach?
    (Mark the box that best fits your answer for each benefit.)

Benefits	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied
Training or access to training courses
Social events or opportunities for socializing
Support, management, and mentorship from my organization
Recognition for my contributions as a peer recovery coach

24. Please rate the following statements based on whether each factor has increased or decreased for you since becoming a peer recovery coach. (Mark the box that best fits your answer for each factor.)

Factors	Increased greatly	Increased	Stayed the same	Decreased	Decreased greatly
My knowledge of risk factors that lead to opioid addiction
My ability to help individuals with opioid addiction
My confidence, self-esteem, or self-management
Skills like teamwork, communication, leadership, or technical skills
My health, well-being, or fitness My sense of community and belonging
My own ability to stay in recovery

25. Do you feel you have adequate support from [grantee org]?

• Yes

• No

• Don’t know

26. Are you currently in long-term recovery?

• Yes

• No

• Prefer not to say

If no, skip to question 30 about your experience as a volunteer.

27. Do you feel possessing lived experience affects relationship-building with your clients?

• Yes

• No

• Prefer not to say

28. Do you feel that being a recovery coach is transforming your own life?

• Yes

• No

• Prefer not to say

29. In your recovery, did you ever have any peer recovery support?

• Yes

• No

• Prefer not to say

30. How important is it that AmeriCorps allows you scheduling flexibility to serve [part-time or near full-time] so that you can attend to your own recovery needs?

• Very important

• Important

• Neither important nor unimportant

• Unimportant

• Very unimportant

Experience as an AmeriCorps volunteer

31. Is this position with AmeriCorps your first professional job experience?

• Yes

• No

32. Do you plan to use the education award?

• Yes

• No

• Don't know

If no, skip to question 34.

33. How do you plan to use the education award?

• To go to community college

• To go to trade school

• To go to a 4-year college

• Other (please specify): ____________

• Don’t know

34. Do you maintain contact with other coaches?

• Yes

• No

If no, skip to question 36.

35. How useful is maintaining contact with other coaches?

• Very useful

• Useful

• Not clear

• Not very useful

36. Do you think this position will provide you with future job opportunities?

• Yes

• No

Demographics

37. What is your age?

• 18-29

• 30-39

• 40-49

• 50-59

• 60-69

• 70-79

• 80+

38. How do you describe your gender? (Select all that apply.)

• Male (including transgender men)

• Female (including transgender women)

• Non-binary/non-conforming

• Prefer to self-describe as ____________

• Prefer not to say

39. Which one of these groups would you say best represents your race?

• White

• Black or African American

• American Indian or Alaska Native

• Asian or Pacific Islander

• Other (please specify): ____________

• Don’t know

• Prefer not to say

40. Are you of Hispanic, Latino/a, or Spanish origin?

• Yes

• No

• Don’t know

• Prefer not to say

41. What is the highest grade or year of school you completed?

• Never attended school or only kindergarten

• Elementary

• Some high school

• High school graduate or equivalent

• Some college or technical school

• College graduate

• Prefer not to say

42. Do you have any other comments? ____________

BENEFICIARY (COMPARISON) SURVEY PROTOCOL

(Questions marked with * are for beneficiaries only.)

Experience with the recovery service

1. How long have you been receiving services from [organization name]?

• Less than one month

• 1-3 months

• 4-6 months

• 7-12 months

• More than a year

2. What type of services do you receive from [organization name]? (Select all that apply.)

• Peer recovery coaching

• Naloxone training

• Outpatient treatment

• Inpatient treatment

• Medication-assisted treatment

• Group sessions

• Individual sessions

• Referrals to services with other organizations (e.g., food assistance, transportation assistance, emergency shelter, treatment services, etc.)

• Other (please specify): ____________

3. Were you connected to outside services, such as housing assistance, health services, food assistance, etc. by [organization name]?

• Yes

• No

• Not applicable

If no, skip to question 5.

4. What kind of outside services were you referred to by [organization name]?

• Emergency shelter

• Food assistance

• Physical addiction-related health services (e.g., detoxification, residential programs, medication assisted therapy, etc.)

• Behavioral addiction-related health services (e.g., group counseling, individual counseling, 12-step programs, etc.)

• Other health services

• Other (please specify): ____________

5. How often do you communicate or check in with your peer recovery coach?*

• Multiple times each day

• Once per day

• One to four times per week

• One or two times per month

• A couple of times per year

• Other (please specify): ____________

6. About how many hours per week do you work with your peer recovery coach?*

• Less than 1 hour

• 1-4 hours

• 5-8 hours

• 9-16 hours

• 17+ hours

• Other (please specify): ____________

7. How do you rate the quality of services you have received with [organization name]?

• Excellent

• Good

• Fair

• Poor

8. Please rate the quality of services your peer recovery coach provides from 1 to 5, with
   1 being lowest quality and 5 being highest quality.*

• 1

• 2

• 3

• 4

• 5

9. My treatment plan reflects my culture and worldview. (Select the response that best fits your answer.)

• Completely

• Very well

• Somewhat

• Not very well

• Not at all

10. How likely are you to recommend this program to another person who uses opioids?

• Very likely

• Likely

• Not likely or unlikely

• Unlikely

• Very unlikely

11. Would you ever consider becoming a peer recovery coach?

• Yes

• No

• Don’t know

• Prefer not to say

Outcomes

12. How often have you used opioids in the last 30 days?

• One or more times per day

• A few times per week

• A few times per month

• Once a month

• I have not used opioids in the last 30 days

13. Since entering recovery, how often have you utilized physical and/or behavioral health services on average?

• Daily

• Once per week or more

• One to two times per month

• A couple of times per year

• Less than two times per year

• I do not attend health services

• Other (please specify): ____________

14. Please rate your level of agreement with each statement below. [answer options for all will be strongly disagree, disagree, neither agree or disagree, agree, and strongly agree]

• There are more important things to me in life than using substances.

• In general, I am happy with my life.

• I have enough energy to complete the tasks I set for myself.

• I am proud of the community I live in and feel a part of it.

• I get the support I need from friends.

• I get the support I need from family.

• I regard my life as challenging and fulfilling without the need for using drugs or alcohol.

• My living space has helped drive my recovery journey.

• Since entering recovery, I take full responsibility for my actions.

• I am happy to deal with a range of professional people.

• I am making progress on my recovery journey.

Demographics

15. What is your age?

• 17 or younger

• 18-29

• 30-39

• 40-49

• 50-59

• 60-69

• 70-79

• 80+

16. How do you describe your gender? (Select all that apply.)

• Male (including transgender men)

• Female (including transgender women)

• Non-binary/non-conforming

• Prefer to self-describe as ____________

• Prefer not to say

17. Which one of these groups would you say best represents your race?

• White

• Black or African American

• American Indian or Alaska Native

• Asian or Pacific Islander

• Other (please specify): ____________

• Don’t know

• Prefer not to say

18. Are you of Hispanic, Latino/a, or Spanish origin?

• Yes

• No

• Don’t know

• Prefer not to say

19. What is the highest grade or year of school you completed?

• Never attended school or only kindergarten

• Elementary

• Some high school

• High school graduate or equivalent

• Some college or technical school

• College graduate

• Prefer not to say

20. Do you have any other comments? ____________

BENEFICIARY INTERVIEW PROTOCOL

1. Let’s begin with some brief introductions. Please tell me a little bit about yourself.

2. If you are comfortable sharing, can you please tell us how you became involved with [organization name] and how long you have been with the program?

3. Please describe what it’s like working with a [peer recovery coach]. What kind of support do you receive from your [peer recovery coach]?

4. How frequently do you communicate with your [peer recovery coach]? Do you reach out to your coach, or does the coach normally initiate communication?

5. Have you worked with the same [peer recovery coach] since you have been with [organization name]?

6. How has COVID impacted you seeking treatment?

7. How effective has the [peer recovery coach model] been on your path to achieve sustained recovery?

(Probe, if considered effective): What in your view makes the [peer recovery coach model] effective?

(Probe, if not considered effective): What in your view makes the [peer recovery coach model] ineffective?

8. In what ways could your interactions with [peer recovery coaches] be improved?

9. What are the most important skills for a [peer recovery coach] to have?

10. Describe how your [peer recovery coach] incorporates your culture and worldview into your treatment plan?

11. What are the most important aspects of your recovery from opioid use disorder that you hope to address with your [peer recovery coach]?

12. How important is it to hear about your [peer recovery coach]’s lived experience and recovery when problem-solving or discussing your own recovery? How often is this
    brought up?

13. Did your [peer recovery coach] or [organization name] connect you with outside resources, such as emergency shelter services, food services, physical or behavioral health providers, or other services?

14. Do you attend health services, including physical or behavioral services, outside of your interactions with [organization name]?

15. Would you say that you attend more or fewer services than when you started working with [organization name]?

16. Does working with a [peer recovery coach] help you reduce your opioid use? If so, how?

17. How has your quality of life changed since beginning your work with a [peer recovery coach]?

18. If you saw someone who was struggling in recovery, would you recommend this program to that person?

19. Would you ever consider becoming a [peer recovery coach] yourself? Why or
    why not?

20. Do you have any additional feedback or insights you would like to share with us regarding the program?

EVALUATION TA PRE–POST OUTCOME SURVEYS

1. Awareness/Attitudes/Motivation

To what extent do you agree with the following statements in A and B? (Using the following scale, fill in each blank with the number that best corresponds to your answer. 1 – Strongly Disagree, 2 – Disagree, 3 – Neither Agree nor Disagree, 4 – Agree, 5 – Strongly Agree) (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

A. I think that an evaluation…

1. Will help me understand my program.

2. Will inform the decisions I make about my program.

3. Will justify funding for my program.

4. Will help to convince managers that changes are needed in my program.

5. Is necessary to improve my program.

6. Should involve program participants in the evaluation process.

7. Will help improve services to target populations.

8. Takes away resources that can be used to provide services.

9. Is incorporated in my daily work.

B. I am motivated to…

1. Learn about evaluation.

2. Start evaluating my program.

3. Support other staff to evaluate their program.

4. Encourage others to buy into evaluating our program.

C. To what extent are the following factors important in your motivation to engage in

program evaluation? (Click “Choose an item” to select the response that best

corresponds to your answer for each one.)

1. A need to meet accountability requirements.

2. A desire to enlighten and support government policymaking and planning.
   

3. Changes in the organization (e.g., reorganization, new leadership, or vision).

4. A mandate from leadership to increase the learning function of evaluation.

5. A perceived lack of evaluation knowledge and skills.

6. A desire to seek new or increased funding.

7. A desire to use evaluation to make program improvements.

8. Other factors (please specify): Click or tap here to enter text.

2. Skills/Knowledge/Behaviors

A. To what extent do you agree with the following statements? (Click “Choose an item to select the response that best corresponds to your answer for each one.)

I know how to…

1. Recognize key concepts in evaluation (evidence, data, performance measurement,

evaluation).

2. Recognize the components of an evaluation plan.

3. Describe what a “theory of change” is.

4. Recognize how theory of change connects to a logic model.

5. Recognize the attributes of a good evaluation question.

6. Identify strategies to collect information from participants.

7. Recognize features of process and outcome indicators of a program.

8. Recognize who should collect data.

9. Recognize when data should be collected.

10. Identify common data analysis terms.

k. Recognize how quantitative and qualitative analysis is performed.

11. Recognize how to use the evaluation findings to improve a program.
    

12. Read an evaluation report and recognize its basic components.

B. To what extent do you agree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

In the past year, I helped my organization…

1. Acquire funding to carry out an evaluation or hire an external evaluator (as part of a grant or through other means).

2. Provide training or technical assistance to conduct evaluation (by staff, consultants, or other means).

3. Conduct evaluations of programs funded by my organization.

4. Use evaluation findings to improve programs funded by my organization.

5. Use evaluation findings to demonstrate and communicate effectiveness of programs funded by my organization.

C. Does your organization have any of the following? (Click “Choose an item” to select “yes” or “no” for each statement.)

1. Staff position(s) or a group within your organization dedicated to evaluation.

2. External evaluation partner(s)—consultant(s) or organization(s) that provide your organization with evaluation services.

3. Part of the organization’s budget dedicated to evaluation.

3. Barriers

To what extent have you experienced the following barriers to conducting a program evaluation? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

1. Not enough time.

2. Not enough money.

3. Not knowing how to conduct a program evaluation.

4. Not having assistance with data collection.

5. Not getting enough people to respond to surveys.

6. Not knowing how to analyze data.

7. Not having people to turn to for consultation and assistance.

8. Not knowing what questions to ask.

9. Not knowing how to write up results.

10. Other factors (please specify): Click or tap here to enter text.

4. About the Respondent

1. [Demographics] What race/ethnic group do you identify with? (Select one.)

☐ Asian/Asian-American

☐ Black/African-American

☐ Hispanic/Latino/Latina

☐ White

☐ Native American/American Indian/Alaskan Native/Native Hawaiian/
Pacific Islander

☐ Multi-racial or multi-ethnic (2+ races/ethnicities)

☐ Prefer not to respond

2. [Demographics] What is your gender? (Select one.)

☐ Male

☐ Female

☐ Non-conforming

☐ Prefer not to respond

☐ Prefer to self-describe (please specify): Click or tap here to enter text.

3. How long have you worked with your organization? (Select one.)

☐ Less than 1 year

☐ 2-5 years

☐ 5-10 years

☐ 10-20 years

☐ 20+ years

4. What is your position in the organization? (Select one.)

☐ Leader (CEO, executive director)

☐ Other executive

☐ Manager/supervisor

☐ Staff

☐ Board member

☐ Other (please specify): Click or tap here to enter text.

5. How long has your organization worked with AmeriCorps (formerly known as CNCS)? (Select one.)

☐ Less than 1 year

☐ 2-5 years

☐ 5-10 years

☐ 10-20 years

☐ 20+ years

EVALUATION TA POST-PRESENTATION SURVEYS

Module 1 — Presentation 1: Introduction and Evaluation Basics

Knowledge Questions

1. Which of the following is NOT a good reason to conduct a program evaluation? (Select one.)

☐ a. Program A wants to demonstrate the benefits of the program to stakeholders.

☐ b. Program B wants to generate a daily report for its employees.

☐ c. Program C wants to understand what needs to be improved in the program.

☐ d. Program D wants to make decisions about resource allocation.

2. Please match the following terms with the right definition. (Click “Choose an item” to select the letter that corresponds to the correct definition for each term.)

   ANSWER	TERMS	DEFINITIONS
   	Evidence	a. The reasons why you think your program will achieve its goals
   	Evaluation	b. Systematic investigation about your program
   	Quantitative data	c. Numerical information that can be counted, quantified, and mathematically analyzed
   	Qualitative data	d. Narrative information, often describing context, motivations, and how and why answers

3. Which of these statements best highlights the difference between program evaluation and performance measurement? (Select one.)

☐ a. Performance measurement uses a systematic approach; evaluation uses

regular updates of changes over time.

☐ b. Performance measurement answers program-level research questions;

evaluation answers routine questions.

☐ c. Performance measurement reports data that require programmatic change;

evaluation reports data that require immediate change.

☐ d. Performance measurement indicates whether a project is moving

according to plan; evaluation systematically examines the workings of a

program.

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ Very Dissatisfied

☐ Dissatisfied

☐ Unsure

☐ Satisfied

☐ Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

c. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

3. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. Why evaluations can be useful.

   2. The difference between evidence and data.

   3. The difference between evaluation and performance measurement.

4. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 1 — Presentation 2: Theory of Change and Logic Models

Knowledge Questions

1. Which of these best defines what a theory of change is? (Select one.)

☐ a. An explanation of any negative or positive impact your program has on the

community

☐ b. An explanation of what your program’s ultimate goal is

☐ c. An example of how you’d like to change your role within your program

☐ d. A reflection on how you achieve your personal goals every day within

your program

2. Which of the following is NOT a reason to develop a theory of change? (Select one.)

☐ a. It ensures people will not ask you to evaluate your program.

☐ b. It forces your organization to articulate the assumptions underlying your

program.

☐ c. It highlights the goals your program is trying to achieve.

☐ d. It shows the unique contribution your program makes to achieve its goals.

3. Select the statement that best describes how a logic model connects to your theory of change. (Select one.)

☐ a. A theory of change visually describes a logic model.

☐ b. A logic model visually shows how your program’s theory of change works.

☐ c. A theory of change visually represents the connections between program

components; a logic model is a statement describing the program’s goals.

☐ d. A logic model presents in one chart every theory of change that you can come

up with.

4. Using what you’ve learned in class about what goes into a logic model and the following theory of change, select one answer in questions a–f based on the following scenario below.

Adults in our city suffer from opioid addiction. We believe this is because adults do not know how to get through the recovery process, and often do not have resources to guide them through recovery. Our theory is that if adults in recovery are provided with recovery coaches to guide them through the process, then the adults will recover from the addiction and not relapse because they will have to be able to develop skills that promote sustainable recovery as well as have someone watching out for them as they navigate the recovery process. Our peer recovery coaches help adults identify and navigate treatment services, obtain employment and housing, access basic resources, and establish new social networks. By helping adults address situational, personal, and medical challenges they may face through a holistic approach, we will ease the burden of situational stressors on adults in recovery. Ultimately, this will reduce the likelihood of relapse.

1. An example of an input for this program would be:

☐ i. Funding for the program

☐ ii. Recruitment of recovery coaches

☐ iii. Number of beneficiaries who obtain housing because of the program

☐ iv. Changes in beneficiaries’ motivation to network

☐ v. Reduced likelihood of relapse for beneficiaries

2. An example of an activity for this program would be:

☐ i. Funding for the program

☐ ii. Recruitment of recovery coaches

☐ iii. Number of beneficiaries who obtain housing because of the program

☐ iv. Changes in beneficiaries’ motivation to network

☐ v. Reduced likelihood of relapse for beneficiaries

3. An example of an output for this program would be:

☐ i. Funding for the program

☐ ii. Recruitment of recovery coaches

☐ iii. Number of volunteers participating in coaching sessions

☐ iv. Changes in beneficiaries’ motivation to network

☐ v. Reduced likelihood of relapse for beneficiaries

4. An example of a short-term outcome for this program would be:

☐ i. Funding for the program

☐ ii. Recruitment of recovery coaches

☐ iii. Number of beneficiaries who obtain housing because of the program

☐ iv. Changes in beneficiaries’ motivation to network

☐ v. Likelihood of relapse for beneficiaries

5. An example of an intermediate outcome for this program would be:

☐ i. Funding for the program

☐ ii. Recruitment of recovery coaches

☐ iii. Number of beneficiaries who obtain housing because of the program

☐ iv. Changes in beneficiaries’ motivation to network

☐ v. Reduced likelihood of relapse for beneficiaries

6. An example of a long-term outcome for this program would be:

☐ i. Funding for the program

☐ ii. Recruitment of recovery coaches

☐ iii. Number of beneficiaries who obtain housing because of the program

☐ iv. Changes in beneficiaries’ motivation to network

☐ v. Reduced likelihood of relapse for beneficiaries

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

3. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. What a theory of change is.

   2. How to come up with a theory of change for my program.

   3. What a logic model is.

   4. What the difference is between a logic model and a theory of change.

5. How to figure out what goes into a logic model.

1. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 1 — Presentation 3: Evaluation Planning

Knowledge Questions

1. Which of the following is an example of a good evaluation question? (Select one.)

☐ a. Does the opioid recovery program work?

☐ b. Are the recovery coaches useful?

☐ c. To what extent does likelihood of relapse differ for clients who participate

in this program compared to others in similar programs without peer recovery coaches?

☐ d. Does our program improve its participants’ happiness?

2. What’s the difference between process and outcome evaluations? (Select one.)

☐ a. Process evaluations examine how a program works; outcome evaluations

measure the results or effects of a program.

☐ b. Process evaluations measure changes in attitudes; outcome evaluations

examine whether the program has sufficient resources.

☐ c. Process evaluations measure changes to beneficiaries; outcome evaluations

measure which pieces of the program are working as expected.

☐ d. Process evaluations measure the results of a program; outcome evaluations

measure program activities.

3. Which of the following is an example of a quantitative indicator for a process evaluation? (Select one.)

☐ a. Students’ perceptions about what did or did not work well in their program

☐ b. AmeriCorps members’ opinions about what they liked and did not like about

their training

☐ c. The number of students who received tutoring

☐ d. Attributes of participants who completed all program components and those

who dropped out

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

c. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

3. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The difference between a process and an outcome evaluation.

   2. The attributes of a good evaluation question.

4. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 — Presentation 1: Preparing to Collect Data

Knowledge Questions

1. Which of the following is an example of primary data that can be collected to evaluate the impact of the recovery coach program? (Select one.)

☐ a. Housing data on clients from the U.S. Census Bureau

☐ b. Hospital records from the National Inpatient Sample

☐ c. Survey responses from clients in the program

☐ d. Employment information from the state

2. When collecting data from the population, which of the following is a good reason to draw a sample? (Select all that apply.)

☐ a. It can be time consuming to collect data from an entire population.

☐ b. It costs less to collect data from a sample than a population.

☐ c. One can get a higher response rate with data collected from a sample than from a population with follow-up efforts.

☐ d. A sample can be representative of a population.

3. When should you collect the following types of data? (Write the following options in the blanks below: before the intervention, during the intervention, after the intervention.)

   1. Baseline data: Click or tap here to enter text. (before the intervention)

   2. Number of times adults have interacted with recovery coaches: Click or tap here to enter text. (during or after the intervention)

   3. Changes in employment: Click or tap here to enter text. (after the intervention)

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

c. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

3. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The difference between primary data and secondary data.

   2. When data should be collected to evaluate a program.

4. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 — Presentation 2: Data Collection Techniques

Knowledge Questions

1. If you are intending to collect qualitative data from a participant, which data collection technique would you NOT utilize? (Select one.)

☐ a. Structured interviews

☐ b. Focus groups

☐ c. Structured observations

☐ d. Closed-ended survey questions

2. Read the questions below and identify the common pitfall (if any) associated with each of the survey questions. (Select one answer for each question.)

   1. Do you feel that you have had a great experience with the recovery coaches in the program?

☐ i. Double-barreled question

☐ ii. Leading question

☐ iii. Social desirability question

☐ iv. Non-applicable question

☐ v. No pitfall

2. Please rate the quality of the monthly meetings with your coach, with 1 being “very useful” to 5 being “not useful at all.”

☐ i. Double-barreled question

☐ ii. Leading question

☐ iii. Social desirability question

☐ iv. Non-applicable question

☐ v. No pitfall

3. Do you have any recommendations or complaints about your recovery coach?

☐ i. Double-barreled question.

☐ ii. Leading question.

☐ iii. Social desirability question.

☐ iv. Non-applicable question.

☐ v. No pitfall.

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

3. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. When to use surveys for primary data collection.

   2. When to use focus groups for primary data collection.

   3. When to use interviews for primary data collection.

4. When to use structured observations for primary data collection.

5. The different types of data collection techniques.

2. What did you enjoy about this learning experience?

Click or tap here to enter text.

2. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 — Presentation 3: Data Analysis

Knowledge Questions

1. For which question would quantitative data be used to answer? (Select one.)

☐ a. How do AmeriCorps recovery coaches interact with clients recovering from

opioid addiction?

☐ b. Why do some clients relapse while others do not?

☐ c. How did program beneficiaries change their behavior after attending coaching

sessions with recovery coaches?

☐ d. How do participants rate their interactions with recovery coaches?

2. Which of the following is an example of qualitative analysis of data? (Select one.)

☐ a. Frequencies

☐ b. Counts

☐ c. Minimum and maximum

☐ d. Interpretation of direct quotes

3. Which of the following is an effective way to quantitively analyze data? (Select one.)

☐ a. Measure whether a difference between two groups is statistically

significant

☐ b. Code participant responses to analyze themes in the data

☐ c. Interpret differences in categories of coded responses

☐ d. Take narrative notes

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

3. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. Quantitative analysis techniques.

   2. Qualitative analysis techniques.

2. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 3 — Presentation 1: Evaluation Reporting

Knowledge Questions

1. Which of these questions should the discussion section of an evaluation report answer? (Select one.)

☐ a. What was the approach used to analyze the data?

☐ b. What was the theory of change for the program?

☐ c. Did the findings of the report support your theory of change?

2. Which of these questions should the methodology section of an evaluation report answer? (Select one.)

☐ a. What was the approach used to analyze the data?

☐ b. What was the theory of change for the program?

☐ c. Did the findings of the report support your theory of change?

3. Which of these questions should the introduction section of an evaluation report answer? (Select one.)

☐ a. What was the approach used to analyze the data?

☐ b. What was the theory of change for the program?

☐ c. Did the findings of the report support your theory of change?

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for
each one.)

1. The information I learned in this module was useful for me.

2. The instructor(s) clearly explained the material covered in this module.

3. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

3. After taking this module, how do you rate your understanding of the following topic? (Click “Choose an item” to select the response that best corresponds to your answer.)

   1. What goes into an evaluation report.

4. What did you enjoy about this learning experience?

Click or tap here to enter text.

3. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 3 — Presentation 2: Using Evaluation for Program Improvement

Knowledge Questions

1. Select all the ways you can use findings from a process evaluation for program improvement. (Select all that apply.)

☐ a. To demonstrate improvements for beneficiaries after coaching sessions

☐ b. To plan further evaluations to examine long-term reductions in

likelihood of relapse

☐ c. To demonstrate changes in beneficiaries’ attitudes after participation in

the program

☐ d. To change the duration of training the recovery coaches receive

☐ e. To revise the intervention model

☐ f. To change the program activities

2. Select all the ways you can use findings from an outcome evaluation for program improvement. (Select all that apply.)

☐ a. To change the program activities

☐ b. To change the duration of training the recovery coaches receive

☐ c. To revise the intervention model

☐ d. To demonstrate improvements for beneficiaries after coaching sessions

☐ e. To plan further evaluations to examine long-term reductions in likelihood

of relapse

☐ f. To demonstrate changes in beneficiaries’ attitudes after participation in

the program

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose any explained the material covered in this module.

   1. I felt comfortable voicing any questions I had in class.

   2. The activities were helpful to my understanding of the material.

   3. I understand how to apply this material to my own program.

3. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. Findings from a process evaluation.

   2. Findings from an outcome evaluation.

4. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 3 — Presentation 3: Developing a Long-Term Research Agenda

Knowledge Questions

1. Which of the following does NOT characterize a long-term research agenda? (Select one.)

☐ a. It should be progressive.

☐ b. Once it is established, it cannot be changed.

☐ c. It should build on what has been done in the past.

☐ d. It should point toward future goals.

2. What needs to be done to build evidence to support the program goals over time?
   (Click “Choose an item” to arrange the following tasks in order of complexity, with “1” being the least complex.)

   1. Build consensus about your overall evaluation goals.

   2. Conduct a process evaluation.

   3. Conduct an outcome evaluation.

   4. Conduct an impact evaluation with a comparison group.

   5. Expand program to multiple sites.

Satisfaction Questions

1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

2. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The information I learned in this module was useful for me.

   2. The instructor(s) clearly explained the material covered in this module.

3. I felt comfortable voicing any questions I had in class.

4. The activities were helpful to my understanding of the material.

5. I understand how to apply this material to my own program.

1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

   1. The key components of a long-term research agenda.

   2. How to develop a long-term research agenda for your program.

2. What did you enjoy about this learning experience?

Click or tap here to enter text.

5. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.