EVALUATION
TECHNICAL ASSISTANCE
POST-PRESENTATION SURVEYS
Module 1 — Presentation 1: Introduction and Evaluation Basics
Knowledge Questions
Which of the following is NOT a good reason to conduct a program evaluation? (Select one.)
☐ a. Program A wants to demonstrate the benefits of the program to stakeholders.
☐ b. Program B wants to generate a daily report for its employees.
☐ c. Program C wants to understand what needs to be improved in the program.
☐ d. Program D wants to make decisions about resource allocation.
2. Please match the following terms with the right definition. (Click “Choose an item” to select the letter that corresponds to the correct definition for each term.)
ANSWER |
TERMS |
DEFINITIONS |
|
Evidence |
a. The reasons why you think your program will achieve its goals |
|
Evaluation |
b.
Systematic investigation about your |
|
Quantitative data |
c. Numerical information that can be counted, quantified, and mathematically analyzed |
|
Qualitative data |
d. Narrative information, often describing context, motivations, and how and why answers |
Which of these statements best highlights the difference between program evaluation and performance measurement? (Select one.)
☐ a. Performance measurement uses a systematic approach; evaluation uses regular updates of changes over time.
☐ b. Performance measurement answers program-level research questions; evaluation answers routine questions.
☐ c. Performance measurement reports data that require programmatic change;
evaluation reports data that require immediate change.
☐ d. Performance measurement indicates whether a project is moving
according to plan; evaluation systematically examines the workings of a
program.
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ Very Dissatisfied
☐ Dissatisfied
☐ Unsure
☐ Satisfied
☐ Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
Why evaluations can be useful.
The difference between evidence and data.
The difference between evaluation and performance measurement.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 1 — Presentation 2: Theory of Change and Logic Models
Knowledge Questions
Which of these best defines what a theory of change is? (Select one.)
☐ a. An explanation of any negative or positive impact your program has on the community
☐ b. An explanation of what your program’s ultimate goal is
☐ c. An example of how you’d like to change your role within your program
☐ d. A reflection on how you achieve your personal goals every day within your program
Which of the following is NOT a reason to develop a theory of change? (Select one.)
☐ a. It ensures people will not ask you to evaluate your program.
☐ b. It forces your organization to articulate the assumptions underlying your
program.
☐ c. It highlights the goals your program is trying to achieve.
☐ d. It shows the unique contribution your program makes to achieve its goals.
Select the statement that best describes how a logic model connects to your theory of change. (Select one.)
☐ a. A theory of change visually describes a logic model.
☐ b. A
logic model visually shows how your program’s theory of
change
works.
☐ c. A theory of change visually represents the connections between program components; a logic model is a statement describing the program’s goals.
☐ d. A logic model presents in one chart every theory of change that you can come up with.
Using what you’ve learned in class about what goes into a logic model and the following theory of change, select one answer in questions a–f based on the following scenario.
There is not enough volunteerism in our state. We believe this is because of a need to expand volunteer capacity in our state. Our theory is that if we can leverage volunteers to achieve operational efficiency in areas they are most needed, then we will create a stronger volunteer infrastructure in our state. The strategic partnerships we have made with Service Enterprise organizations help with effectively using volunteers’ skills to address community needs; strengthen the volunteer network by promoting connections between non-profits, corporations, and volunteers; and train volunteers for additional skills they will need. By strengthening capacity building efforts and creating a volunteer network in the state, we will effectively use volunteers to enable organizations to accomplish their missions. Ultimately, this will increase our state’s volunteerism rates.
An example of an input for this program would be:
☐ i. Funding for the program
☐ ii. Trainings provided to volunteers
☐ iii. Number of registered volunteers in the state
☐ iv. Expansion of volunteers’ skill sets
☐ v. Volunteerism rates
An example of an activity for this program would be:
☐ i. Funding for the program
☐ ii. Trainings provided to volunteers
☐ iii. Number of registered volunteers in the state
☐ iv. Expansion of volunteers’ skill sets
☐ v. Volunteerism rates
An example of an output for this program would be:
☐ i. Funding for the program
☐ ii. Trainings provided to volunteers
☐ iii. Number of registered volunteers in the state
☐ iv. Expansion of volunteers’ skill sets
☐ v. Volunteerism rates
An example of a short-term outcome for this program would be:
☐ i. Funding for the program
☐ ii. Trainings provided to volunteers
☐ iii. Number of registered volunteers in the state
☐ iv. Expansion of volunteers’ skill sets
☐ v. Volunteerism rates
An example of an intermediate outcome for this program would be:
☐ i. Funding for the program
☐ ii. Increased number of organizations that report increased effectiveness
☐ iii. Number of registered volunteers in the state
☐ iv. Expansion of volunteers’ skill sets
☐ v. Volunteerism rates
An example of a long-term outcome for this program would be:
☐ i. Funding for the program
☐ ii. Increased number of organizations that report increased effectiveness
☐ iii. Number of registered volunteers in the state
☐ iv. Expansion of volunteers’ skill sets
☐ v. Volunteerism rates
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
What a theory of change is.
How to come up with a theory of change for my program.
What a logic model is.
What the difference is between a logic model and a theory of change.
How to figure out what goes into a logic model.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 1 — Presentation 3: Evaluation Planning
Knowledge Questions
Which of the following is an example of a good evaluation question? (Select one.)
☐ a. Does the volunteer generation program work?
☐ b. Are Service Enterprise Hubs useful?
☐ c. To what extent do volunteerism rates differ for organizations
participating in this program compared to other organizations that did not participate?
☐ d. Does our program improve its volunteers’ skills?
What’s the difference between process and outcome evaluations? (Select one.)
☐ a. Process evaluations examine how a program works; outcome
evaluations measure the results or effects of a program.
☐ b. Process evaluations measure changes in attitudes; outcome evaluations
examine whether the program has sufficient resources.
☐ c. Process evaluations measure changes to beneficiaries; outcome
evaluations measure which pieces of the program are working as expected.
☐ d. Process evaluations measure the results of a program; outcome evaluations
measure program activities.
Which of the following is an example of a quantitative indicator for a process
evaluation? (Select one.)
☐ a. Students’ perceptions about what did or did not work well in their program
☐ b. AmeriCorps members’ opinions about what they liked and did not like
about their training
☐ c. The number of students who received tutoring
☐ d. Attributes of participants who completed all program components and
those who dropped out
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The difference between a process and an outcome evaluation.
The attributes of a good evaluation question.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 2 — Presentation 1: Preparing to Collect Data
Knowledge Questions
Which of the following is an example of primary data that can be collected to evaluate the impact of the recovery coach program? (Select one.)
☐ a. High school graduation rates from the U.S. Census Bureau
☐ b. Hospital records from the National Inpatient Sample
☐ c. Survey responses from volunteers in the program
☐ d. Employment information from the state
When collecting data from the population, which of the following is a good reason to draw a sample? (Select all that apply.)
☐ a. It can be time consuming to collect data from an entire population.
☐ b. It costs less to collect data from a sample than a population.
☐ c. One can get a higher response rate with data collected from a sample
than from a population with follow-up efforts.
☐ d. A sample can be representative of a population.
When should you collect the following types of data? (Fill in each box using the following options: before the intervention, during the intervention, after the intervention.)
a. Baseline data: Click or tap here to enter text. (before the intervention)
Number of times adults have interacted with recovery coaches:
Click or tap here to enter text. (during or after the intervention)
Changes in employment: Click or tap here to enter text. (after the intervention)
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The difference between primary data and secondary data.
When data should be collected to evaluate a program.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 2 — Presentation 2: Data Collection Techniques
Knowledge Questions
If you are intending to collect qualitative data from a participant, which data collection technique would you NOT utilize? (Select one.)
☐ a. Structured interviews
☐ b. Focus groups
☐ c. Structured observations
☐ d. Closed-ended survey questions
2. Read the questions below and identify the common pitfall (if any) associated with each of the survey questions. (Select one answer for each question.)
Do you feel that you have had a great experience with the recovery coaches in the program?
☐ i. Double-barreled question
☐ ii. Leading question
☐ iii. Social desirability question
☐ iv. Non-applicable question
☐ v. No pitfall
Please rate the quality of the monthly meetings with your coach, with 1 being “very useful” to 5 being “not useful at all.”
☐ i. Double-barreled question
☐ ii. Leading question
☐ iii. Social desirability question
☐ iv. Non-applicable question
☐ v. No pitfall
Do you have any recommendations or complaints about your recovery coach?
☐ i. Double-barreled question
☐ ii. Leading question
☐ iii. Social desirability question
☐ iv. Non-applicable question
☐ v. No pitfall
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
When to use surveys for primary data collection.
When to use focus groups for primary data collection.
When to use interviews for primary data collection.
When to use structured observations for primary data collection.
The different types of data collection techniques.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 2 — Presentation 3: Data Analysis
Knowledge Questions
For which question would quantitative data be used to answer? (Select one.)
☐ a. How many volunteers use the volunteer database daily?
☐ b. What do volunteers say about their experience with volunteer trainings?
☐ c. Why are some organizations better at using volunteers for COVID-19 recovery work versus others?
☐ d. What makes it harder for states to provide volunteers to rural areas?
Which of the following is an example of qualitative analysis of data? (Select one.)
☐ a. Frequencies
☐ b. Counts
☐ c. Minimum and maximum
☐ d. Interpretation of direct quotes
Which of the following is an effective way to quantitively analyze data? (Select one.)
☐ a. Measure whether a difference between two groups is statistically significant
☐ b. Code participant responses to analyze themes in the data
☐ c. Interpret differences in categories of coded responses
☐ d. Take narrative notes
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
Quantitative analysis techniques.
Qualitative analysis techniques.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 3 — Presentation 1: Evaluation Reporting
Knowledge Questions
Which of these questions should the discussion section of an evaluation report answer? (Select one.)
☐ a. What was the approach used to analyze the data?
☐ b. What was the theory of change for the program?
☐ c. Did the findings of the report support your theory of change?
Which of these questions should the methodology section of an evaluation report answer? (Select one.)
☐ a. What was the approach used to analyze the data?
☐ b. What was the theory of change for the program?
☐ c. Did the findings of the report support your theory of change?
Which of these questions should the introduction section of an evaluation report answer? (Select one.)
☐ a. What was the approach used to analyze the data?
☐ b. What was the theory of change for the program?
☐ c. Did the findings of the report support your theory of change?
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topic? (Click “Choose an item” to select the response that best corresponds to your answer.)
What goes into an evaluation report.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 3 — Presentation 2: Using Evaluation for Program Improvement
Knowledge Questions
Select all the ways you can use findings from a process evaluation for program improvement. (Select all that apply.)
☐ a. To demonstrate improvement for organizations after training
☐ b. To plan further evaluations to examine long-term increases in rates of
volunteerism
☐ c. To demonstrate changes in organizations’ attitudes after participation in
statewide volunteerism programs
☐ d. To change the duration of training the volunteers receive
☐ e. To revise the intervention model
☐ f. To change the program activities
Select all the ways you can use findings from an outcome evaluation for program improvement. (Select all that apply.)
☐ a. To demonstrate improvements for organizations after training
☐ b. To plan further evaluations to examine long-term increases in rates of
volunteerism
☐ c. To demonstrate changes in organizations’ attitudes after participation in
statewide volunteerism programs
☐ d. To change the duration of training the volunteers receive
☐ e. To revise the intervention model
☐ f. To change the program activities
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
Findings from a process evaluation.
Findings from an outcome evaluation.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
Module 3 — Presentation 3: Developing a Long-Term Research Agenda
Knowledge Questions
Which of the following does NOT characterize a long-term research agenda? (Select one.)
☐ a. It should be progressive.
☐ b. Once it is established, it cannot be changed.
☐ c. It should build on what has been done in the past.
☐ d. It should point toward future goals.
What needs to be done to build evidence to support the program goals over time? (Click “Choose an item” to arrange the following tasks in order of complexity, with “1” being the least complex.)
a. Build consensus about your overall evaluation goals.
b. Conduct a process evaluation.
c. Conduct an outcome evaluation.
d. Conduct an impact evaluation with a comparison group.
e. Expand program to multiple sites.
Satisfaction Questions
Overall, how satisfied are you with this module? (Select one.)
☐ a. Very Dissatisfied
☐ b. Dissatisfied
☐ c. Unsure
☐ d. Satisfied
☐ e. Very Satisfied
How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)
The key components of a long-term research agenda.
How to develop a long-term research agenda for your program.
What did you enjoy about this learning experience?
Click or tap here to enter text.
Do you have any suggestions for how to improve the learning experience?
Click or tap here to enter text.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Zhang, Xiaodong |
| File Modified | 0000-00-00 |
| File Created | 2021-10-07 |