7 Evaluation TA Post Survey

AmeriCorps Program Life Cycle Evaluation—Volunteer Generation Fund Grant Program Evaluation

20211006 Instrument 7 Evaluation Technical Assistance Post Survey

OMB: 3045-0195

Document [docx]
Download: docx | pdf

EVALUATION TECHNICAL ASSISTANCE
POST-PRESENTATION SURVEYS

Module 1 Presentation 1: Introduction and Evaluation Basics

Knowledge Questions

  1. Which of the following is NOT a good reason to conduct a program evaluation? (Select one.)

a. Program A wants to demonstrate the benefits of the program to stakeholders.

b. Program B wants to generate a daily report for its employees.

c. Program C wants to understand what needs to be improved in the program.

d. Program D wants to make decisions about resource allocation.

2. Please match the following terms with the right definition. (Click “Choose an item” to select the letter that corresponds to the correct definition for each term.)

ANSWER

TERMS

DEFINITIONS

Evidence

a. The reasons why you think your program will achieve its goals

Evaluation

b. Systematic investigation about your
program

Quantitative data

c. Numerical information that can be counted, quantified, and mathematically analyzed

Qualitative data

d. Narrative information, often describing context, motivations, and how and why answers

  1. Which of these statements best highlights the difference between program evaluation and performance measurement? (Select one.)

a. Performance measurement uses a systematic approach; evaluation uses regular updates of changes over time.

b. Performance measurement answers program-level research questions; evaluation answers routine questions.

c. Performance measurement reports data that require programmatic change;

evaluation reports data that require immediate change.

d. Performance measurement indicates whether a project is moving

according to plan; evaluation systematically examines the workings of a

program.



Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

Very Dissatisfied

Dissatisfied

Unsure

Satisfied

Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    5. I understand how to apply this material to my own program.

  2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Why evaluations can be useful.

    2. The difference between evidence and data.

    3. The difference between evaluation and performance measurement.

  1. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.


Module 1 Presentation 2: Theory of Change and Logic Models

Knowledge Questions

  1. Which of these best defines what a theory of change is? (Select one.)

a. An explanation of any negative or positive impact your program has on the community

b. An explanation of what your program’s ultimate goal is

c. An example of how you’d like to change your role within your program

d. A reflection on how you achieve your personal goals every day within your program

  1. Which of the following is NOT a reason to develop a theory of change? (Select one.)

a. It ensures people will not ask you to evaluate your program.

b. It forces your organization to articulate the assumptions underlying your

program.

c. It highlights the goals your program is trying to achieve.

d. It shows the unique contribution your program makes to achieve its goals.

  1. Select the statement that best describes how a logic model connects to your theory of change. (Select one.)

a. A theory of change visually describes a logic model.

b. A logic model visually shows how your program’s theory of
change works.

c. A theory of change visually represents the connections between program components; a logic model is a statement describing the program’s goals.

d. A logic model presents in one chart every theory of change that you can come up with.

  1. Using what you’ve learned in class about what goes into a logic model and the following theory of change, select one answer in questions af based on the following scenario.


There is not enough volunteerism in our state. We believe this is because of a need to expand volunteer capacity in our state. Our theory is that if we can leverage volunteers to achieve operational efficiency in areas they are most needed, then we will create a stronger volunteer infrastructure in our state. The strategic partnerships we have made with Service Enterprise organizations help with effectively using volunteers’ skills to address community needs; strengthen the volunteer network by promoting connections between non-profits, corporations, and volunteers; and train volunteers for additional skills they will need. By strengthening capacity building efforts and creating a volunteer network in the state, we will effectively use volunteers to enable organizations to accomplish their missions. Ultimately, this will increase our state’s volunteerism rates.

    1. An example of an input for this program would be:

i. Funding for the program

ii. Trainings provided to volunteers

iii. Number of registered volunteers in the state

iv. Expansion of volunteers’ skill sets

v. Volunteerism rates

    1. An example of an activity for this program would be:

i. Funding for the program

ii. Trainings provided to volunteers

iii. Number of registered volunteers in the state

iv. Expansion of volunteers’ skill sets

v. Volunteerism rates

    1. An example of an output for this program would be:

i. Funding for the program

ii. Trainings provided to volunteers

iii. Number of registered volunteers in the state

iv. Expansion of volunteers’ skill sets

v. Volunteerism rates

    1. An example of a short-term outcome for this program would be:

i. Funding for the program

ii. Trainings provided to volunteers

iii. Number of registered volunteers in the state

iv. Expansion of volunteers’ skill sets

v. Volunteerism rates

    1. An example of an intermediate outcome for this program would be:

i. Funding for the program

ii. Increased number of organizations that report increased effectiveness

iii. Number of registered volunteers in the state

iv. Expansion of volunteers’ skill sets

v. Volunteerism rates

    1. An example of a long-term outcome for this program would be:

i. Funding for the program

ii. Increased number of organizations that report increased effectiveness

iii. Number of registered volunteers in the state

iv. Expansion of volunteers’ skill sets

v. Volunteerism rates

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. What a theory of change is.

    2. How to come up with a theory of change for my program.

    3. What a logic model is.

    4. What the difference is between a logic model and a theory of change.

    1. How to figure out what goes into a logic model.

  1. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.




Module 1 Presentation 3: Evaluation Planning

Knowledge Questions

  1. Which of the following is an example of a good evaluation question? (Select one.)

a. Does the volunteer generation program work?

b. Are Service Enterprise Hubs useful?

c. To what extent do volunteerism rates differ for organizations

participating in this program compared to other organizations that did not participate?

d. Does our program improve its volunteers’ skills?

  1. What’s the difference between process and outcome evaluations? (Select one.)

a. Process evaluations examine how a program works; outcome

evaluations measure the results or effects of a program.

b. Process evaluations measure changes in attitudes; outcome evaluations

examine whether the program has sufficient resources.

c. Process evaluations measure changes to beneficiaries; outcome

evaluations measure which pieces of the program are working as expected.

d. Process evaluations measure the results of a program; outcome evaluations

measure program activities.

  1. Which of the following is an example of a quantitative indicator for a process

evaluation? (Select one.)

a. Students’ perceptions about what did or did not work well in their program

b. AmeriCorps members’ opinions about what they liked and did not like

about their training

c. The number of students who received tutoring

d. Attributes of participants who completed all program components and

those who dropped out

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

  1. The instructor(s) clearly explained the material covered in this module.

  1. I felt comfortable voicing any questions I had in class.

  2. The activities were helpful to my understanding of the material.

  3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The difference between a process and an outcome evaluation.

    2. The attributes of a good evaluation question.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.




Module 2 Presentation 1: Preparing to Collect Data

Knowledge Questions

  1. Which of the following is an example of primary data that can be collected to evaluate the impact of the recovery coach program? (Select one.)

a. High school graduation rates from the U.S. Census Bureau

b. Hospital records from the National Inpatient Sample

c. Survey responses from volunteers in the program

d. Employment information from the state

  1. When collecting data from the population, which of the following is a good reason to draw a sample? (Select all that apply.)

a. It can be time consuming to collect data from an entire population.

b. It costs less to collect data from a sample than a population.

c. One can get a higher response rate with data collected from a sample

than from a population with follow-up efforts.

d. A sample can be representative of a population.

  1. When should you collect the following types of data? (Fill in each box using the following options: before the intervention, during the intervention, after the intervention.)

a. Baseline data: Click or tap here to enter text. (before the intervention)

  1. Number of times adults have interacted with recovery coaches:

Click or tap here to enter text. (during or after the intervention)

  1. Changes in employment: Click or tap here to enter text. (after the intervention)

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The difference between primary data and secondary data.

    2. When data should be collected to evaluate a program.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 Presentation 2: Data Collection Techniques

Knowledge Questions

  1. If you are intending to collect qualitative data from a participant, which data collection technique would you NOT utilize? (Select one.)

a. Structured interviews

b. Focus groups

c. Structured observations

d. Closed-ended survey questions

2. Read the questions below and identify the common pitfall (if any) associated with each of the survey questions. (Select one answer for each question.)

    1. Do you feel that you have had a great experience with the recovery coaches in the program?

i. Double-barreled question

ii. Leading question

iii. Social desirability question

iv. Non-applicable question

v. No pitfall

    1. Please rate the quality of the monthly meetings with your coach, with 1 being “very useful” to 5 being “not useful at all.”

i. Double-barreled question

ii. Leading question

iii. Social desirability question

iv. Non-applicable question

v. No pitfall











  1. Do you have any recommendations or complaints about your recovery coach?

i. Double-barreled question

ii. Leading question

iii. Social desirability question

iv. Non-applicable question

v. No pitfall

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. When to use surveys for primary data collection.

    2. When to use focus groups for primary data collection.

    3. When to use interviews for primary data collection.

    4. When to use structured observations for primary data collection.

    5. The different types of data collection techniques.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 Presentation 3: Data Analysis

Knowledge Questions

  1. For which question would quantitative data be used to answer? (Select one.)

a. How many volunteers use the volunteer database daily?

b. What do volunteers say about their experience with volunteer trainings?

c. Why are some organizations better at using volunteers for COVID-19 recovery work versus others?

d. What makes it harder for states to provide volunteers to rural areas?

  1. Which of the following is an example of qualitative analysis of data? (Select one.)

a. Frequencies

b. Counts

c. Minimum and maximum

d. Interpretation of direct quotes

  1. Which of the following is an effective way to quantitively analyze data? (Select one.)

a. Measure whether a difference between two groups is statistically significant

b. Code participant responses to analyze themes in the data

c. Interpret differences in categories of coded responses

d. Take narrative notes

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied



  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    5. I understand how to apply this material to my own program.

  2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Quantitative analysis techniques.

    2. Qualitative analysis techniques.

  3. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.



Module 3 Presentation 1: Evaluation Reporting

Knowledge Questions

  1. Which of these questions should the discussion section of an evaluation report answer? (Select one.)

a. What was the approach used to analyze the data?

b. What was the theory of change for the program?

c. Did the findings of the report support your theory of change?

  1. Which of these questions should the methodology section of an evaluation report answer? (Select one.)

a. What was the approach used to analyze the data?

b. What was the theory of change for the program?

c. Did the findings of the report support your theory of change?

  1. Which of these questions should the introduction section of an evaluation report answer? (Select one.)

a. What was the approach used to analyze the data?

b. What was the theory of change for the program?

c. Did the findings of the report support your theory of change?

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    1. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topic? (Click “Choose an item” to select the response that best corresponds to your answer.)

    1. What goes into an evaluation report.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.


Module 3 Presentation 2: Using Evaluation for Program Improvement

Knowledge Questions

  1. Select all the ways you can use findings from a process evaluation for program improvement. (Select all that apply.)

a. To demonstrate improvement for organizations after training

b. To plan further evaluations to examine long-term increases in rates of

volunteerism

c. To demonstrate changes in organizations’ attitudes after participation in

statewide volunteerism programs

d. To change the duration of training the volunteers receive

e. To revise the intervention model

f. To change the program activities

  1. Select all the ways you can use findings from an outcome evaluation for program improvement. (Select all that apply.)

a. To demonstrate improvements for organizations after training

b. To plan further evaluations to examine long-term increases in rates of

volunteerism

c. To demonstrate changes in organizations’ attitudes after participation in

statewide volunteerism programs

d. To change the duration of training the volunteers receive

e. To revise the intervention model

f. To change the program activities

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied



  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    5. I understand how to apply this material to my own program.

  2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Findings from a process evaluation.

    2. Findings from an outcome evaluation.

  3. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.




Module 3 Presentation 3: Developing a Long-Term Research Agenda

Knowledge Questions

  1. Which of the following does NOT characterize a long-term research agenda? (Select one.)

a. It should be progressive.

b. Once it is established, it cannot be changed.

c. It should build on what has been done in the past.

d. It should point toward future goals.

  1. What needs to be done to build evidence to support the program goals over time? (Click “Choose an item” to arrange the following tasks in order of complexity, with “1” being the least complex.)

a. Build consensus about your overall evaluation goals.

b. Conduct a process evaluation.

c. Conduct an outcome evaluation.

d. Conduct an impact evaluation with a comparison group.

e. Expand program to multiple sites.

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

a. Very Dissatisfied

b. Dissatisfied

c. Unsure

d. Satisfied

e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.


    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The key components of a long-term research agenda.

    2. How to develop a long-term research agenda for your program.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.



File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorZhang, Xiaodong
File Modified0000-00-00
File Created2021-11-15

© 2024 OMB.report | Privacy Policy