Climate Change Bundle
Evaluation Instruments for AmeriCorps
Submitted to:
Washington,
DC 20024 Submitted
by: Reston,
VA 20190 November
2022
Climate Change
Evaluation Bundling
AmeriCorps, 250 E St., SW
ICF
Incorporated, LLC 1902 Reston Metro Plz
https://www.icf.com
AMERICORPS MEMBER SURVEY PROTOCOL 2
AMERICORPS PROJECT DIRECTOR SURVEY 10
AMERICORPS COMMUNITY PARTNER SURVEY 17
GRANTEE/SPONSOR ORGANIZATION – PROJECT DIRECTOR IN-DEPTH INTERVIEW (60 minutes) 23
GRANTEE/SPONSOR ORGANIZATION – PROGRAM STAFF IN-DEPTH INTERVIEW (60 minutes) 29
AMERICORPS MEMBER FOCUS GROUP (90 minutes) 34
COMMUNITY MEMBER FOCUS GROUP (60 minutes) 39
PARTNER ORGANIZATION IN-DEPTH INTERVIEW (60 minutes) 42
EVALUATION CAPACITY BUILDING PRE-/POST-PROGRAM ASSESSMENT 46
EVALUATION CAPACITY BUILDING PRESENTATION FEEDBACK SURVEYS 51
EVALUATION CAPACITY BUILDING OBSERVATION FORM 74
AmeriCorps Member Survey Consent
Participation
Thank you for taking the time to participate in this survey. This survey is part of a study being conducted by ICF to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your program’s activities and effect on the community you serve. The survey will take about 20 minutes to complete.
Risks
We do not anticipate any risks in participating in this survey. Participation in this survey is voluntary. You can skip any question or stop answering questions at any time.
Your responses to the survey will be kept confidential. Your answers will not affect your current or future work with AmeriCorps. A report summarizing feedback across different organizations and AmeriCorps members will be shared with AmeriCorps and with grantees. The information that we use from this survey will not be identified with any one individual. Only ICF staff and the AmeriCorps Office of Research and Evaluation will have access to the de-identified survey data.
Questions
Should you have any questions about the survey, you may contact study representative Dr. Kimberly Good at [email protected] or Institutional Review Board representative [name].
If you agree to participate in the survey, please acknowledge below by selecting, “I agree to participate.”
I agree to participate.
I do not agree to participate. [If selected, will use skip logic take respondent to the end of the survey]
Background
This is a list of reasons people have for giving their time and efforts to others. Please indicate how important each of the following reasons are for you personally to volunteer.
|
Not important |
Slightly important |
Important |
Very important |
Don’t know |
It provides an opportunity to use your knowledge, skills, and experience. |
|
|
|
|
|
It is something that will help your own neighborhood or community. |
|
|
|
|
|
It is a way to make a difference for a cause or issue that is important to you OR in which you are interested. |
|
|
|
|
|
It is a way to meet people or gather with people. |
|
|
|
|
|
Because you feel a personal responsibility to help others when they need it. |
|
|
|
|
|
It makes your own life more satisfying. |
|
|
|
|
|
It lets you learn things through direct, hands-on experience. |
|
|
|
|
|
It helps you to explore possible career options. |
|
|
|
|
|
Source: These items are adapted from the Civic Engagement Volunteer Experience Survey and the Volunteer Motivation Scale.
Current Role and Services Provided
How long have you been with [grantee/sponsor name] as a national service member?
About how many hours per week do you serve as a member?
Less than 5 hours
5–10 hours
11–20 hours
21–30 hours
More than 30 hours
What types of services do you provide that are designed to impact climate change? Please select all of the services that your organization provides and indicate how long you have been providing the service.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For services connected to climate change that were selected in item 4, please describe (in greater detail) the activities that are being implemented to address climate change? ________________
________________________________________________________________________________
Why did you choose to serve in [program name]? _______________________________________
How would you describe the community that you are serving?
City
Suburb
Town
Rural area
Are you a member of the specific geographic community that you are serving?
Yes
No
Training
In general, how many hours of training have you received to prepare you to serve as a member?
Less than 1 hour
1–4 hours
5–8 hours
9–16 hours
17+
hours
What was the content of the training? (Select all that apply.)
Environmental sustainability (e.g., water conservation, reducing energy consumption)
Renewable energy (e.g., wind or solar power)
Technical skills (e.g., using new technology or tools)
Communication skills (e.g., verbal communication, active listening)
Problem solving skills (e.g., identifying a problem, exploring alternative solutions)
Teamwork skills (e.g., team building, conflict resolution)
Career preparation
Other (please specify): ________________________
How helpful was the training for your role as a national service member in this program?
Very helpful
Helpful
Somewhat helpful
Not at all helpful
Program Outcomes
In the last 6 months, how often, if at all, have you engaged in the following climate change activities and actions. Please indicate the level to which you have participated on a scale from “Never” engaged to “Frequently” engaged.
|
Never |
Rarely |
Sometimes |
Frequently |
Became involved with an environmental group or political party (e.g., volunteer, summer job, etc.). |
|
|
|
|
Participated in an educational event (e.g., workshop) related to the environment. |
|
|
|
|
Participated in a community event that focused on environmental awareness. |
|
|
|
|
Participated in nature conservation efforts (e.g., planting trees, restoration of waterways). |
|
|
|
|
Spent time working with a group/organization that deals with the connection of the environment to other societal issues such as justice or poverty. |
|
|
|
|
Source: These items are adapted from the Environmental Action Scale.
Listed below are actions that one could take in the future to help reduce climate change. Please use the rating scale below to indicate how likely you would be to take the action.
|
Definitely won’t |
Probably won’t |
Probably will |
Definitely will |
Vote on issues related to climate change. |
|
|
|
|
Organize an educational event (e.g., workshop) related to climate change issues. |
|
|
|
|
Use social media to raise awareness about climate change issues. |
|
|
|
|
Personally write to or call a politician/ government official about climate change issues. |
|
|
|
|
Organize a petition for an environmental cause. |
|
|
|
|
Organize a community event to fight climate change. |
|
|
|
|
Organize a community event to promote environmental justice. |
|
|
|
|
Source: These items are adapted from the Environmental Action Scale.
The following items pertain to the impact of the program on your future education and career goals. Please indicate the extent to which participation in the program affected the following.
|
Not at all |
Very little |
Somewhat |
A great deal |
Participation in this program enhanced my general employment skills (e.g., communication, problem solving, teamwork). |
|
|
|
|
Participation in this program increased my ability to use new technology/tools. |
|
|
|
|
Participation in this program improved my ability to work with people from different racial, ethnic, social, or educational backgrounds. |
|
|
|
|
Participation in this program impacted the career/occupation I will pursue in the future. |
|
|
|
|
Participation in this program influenced the degree/major I will pursue. |
|
|
|
|
Do you have any interest in obtaining a job related to climate change?
Yes
No If
no, go to item 17.
If yes, please rate your willingness to pursue a career in the following job categories?
|
Definitely won’t |
Probably won’t |
Probably will |
Definitely will |
Agriculture and natural resource conservation |
|
|
|
|
Clean transportation and fuels |
|
|
|
|
Disaster services |
|
|
|
|
Education, compliance, and awareness |
|
|
|
|
Energy efficient product manufacturing, distribution, construction, installation, or maintenance |
|
|
|
|
Natural and sustainable product manufacturing |
|
|
|
|
Pollution prevention or environmental cleanup |
|
|
|
|
Other (explain below) |
|
|
|
|
If other was selected in item 16, please specify: ___________________________________________
The items in this section are designed to measure your perceptions of the effectiveness of [program name]. Please indicate the level at which you think the program is effective at addressing the following in the community. (If you do not have enough information to respond to the item, please select “Don’t know.”)
|
Not Effective |
Somewhat Effective |
Effective |
Very Effective |
Don’t Know |
Increase the use of renewable energy sources |
|
|
|
|
|
Improve energy efficiency |
|
|
|
|
|
Prepare for extreme weather and climate |
|
|
|
|
|
Prepare for, respond to, or recover from natural disasters |
|
|
|
|
|
Preserve public lands and/or waterways |
|
|
|
|
|
Protect and/or restore biodiversity |
|
|
|
|
|
Help the community become more resilient to a changing climate |
|
|
|
|
|
Increase community engagement in climate change activities |
|
|
|
|
|
Increase self-sufficiency among community members |
|
|
|
|
|
Focus on the needs of vulnerable populations (economic, health, housing, transportation, social) |
|
|
|
|
|
Increase community members’ awareness of resources to meet their basic needs (e.g., food, shelter, medical) |
|
|
|
|
|
Increase community members’ knowledge about climate change |
|
|
|
|
|
Increase community members’ positive attitudes toward activities that mitigate climate change |
|
|
|
|
|
Increase community members’ behaviors around climate change |
|
|
|
|
|
Recruit diverse program participants |
|
|
|
|
|
Support local workforce development needs |
|
|
|
|
|
Increase skills development toward future employment |
|
|
|
|
|
Increase skills development toward green jobs |
|
|
|
|
|
Demographics
What is your age?
Under 18
18–29
30–39
40–49
50–59
60–69
70–79
80+
How do you describe your gender? (Select all that apply.)
Male (including transgender men)
Female (including transgender women)
Nonbinary/nonconforming
Prefer to self-describe as ____________
Prefer not to say
Which one of these groups would you say best represents your race?
White
Black or African American
American Indian or Alaska Native
Asian
Native Hawaiian or Pacific Islander
Multi-racial or multi-ethnic (2+ races/ethnicities)
Other (please specify): ____________
Don’t know
Prefer not to say
Are you of Hispanic, Latino/a, or Spanish origin?
Yes
No
Don’t know
Prefer not to say
What is the highest grade or year of school you completed?
Never attended school or only kindergarten
Elementary school
Some high school
High school graduate or equivalent
Some college or technical school (If selected, go to item 23)
College graduate (If selected, go to item 23)
Some graduate school (If selected, go to item 23)
Completed graduate school (If selected, go to item 23)
Prefer not to say
What was your major area of study? ____________________________________________________
Did you grow up in an area that is similar to the community that you are serving?
Yes If yes, go to item 26.
No
How would you describe the community where you grew up?
City
Suburb
Town
Rural area
Do you have any other comments? _____________________
Grantee/Sponsor Project Director Survey Consent
Participation
Thank you for taking the time to participate in this survey. This survey is part of a study being conducted by ICF on behalf of AmeriCorps and [program/project name] to better understand how their programs/projects are working. More specifically, we are interested in your program/project’s activities and effect on the community you serve. The survey will take about 20 minutes to complete.
Risks
We do not anticipate any risks in participating in this survey. Participation in this survey is voluntary. You can skip any question or stop answering questions at any time.
Your responses to the survey will be kept confidential. Your answers will not affect your current or future work with AmeriCorps. A report summarizing feedback across different organizations and AmeriCorps members will be shared with AmeriCorps. The information that we use from this survey will not be identified with any one individual. Only ICF staff and the AmeriCorps Office of Research and Evaluation will have access to the de-identified survey data.
Questions
Should you have any questions about the survey, you may contact study Dr. Kimberly Good at [email protected] or Institutional Review Board representative [name].
If you agree to participate in the survey, please acknowledge below by selecting, “I agree to participate.”
I agree to participate.
I do not agree to participate. [If selected, will use skip logic take respondent to the end of the survey]
Background
What is your current role at [organization name]?
Program assistant
Program manager
Director
Other (please specify): ____________
How long have you been with [organization name]?
Less than 1 year
1–5 years
6–10 years
11–15 years
16+ years
How long have you been with [organization name] in your current role?
Less than 1 year
1–5 years
6–10 years
11–15 years
16+ years
How many national service members does your program oversee? ____________
National Service Member Training
How many hours of training do national service members receive?
Less than 1 hour
1–4 hours
5–8 hours
9–16 hours
17+ hours
What is the content of the training? (Select all that apply.)
Environmental sustainability (e.g., water conservation, reducing energy consumption)
Renewable energy (e.g., wind or solar power)
Technical skills (e.g., using new technology or tools)
Communication skills (e.g., verbal communication, active listening)
Problem solving skills (e.g., identifying a problem, exploring alternative solutions)
Teamwork skills (e.g., team building, conflict resolution)
Career preparation
Other (please specify): ________________________
Services and Activities
In general, what types of services does your organization provide? (Select all that apply.)
Education (e.g., tutoring, literacy, education for children and youth)
Health (e.g., nutrition, access to care, prevention, awareness, mental health)
Services to veterans, members of the armed forces, or their families
Environment (e.g., education on environmental issues, land conservation, energy conservation, ecosystem development/maintenance)
Disaster services (e.g., disaster education, preparation, mitigation, response)
Economic opportunity (e.g., housing services, employment counseling, job training, financial literacy, financial assistance)
Other
services (please specify): ____________________________________
Who does your organization serve? Please select the populations that your organization serves. (Select all that apply.)
Formerly Incarcerated
Homeless
Immigrant (documented and undocumented)
Individuals with disabilities
Low income
Non-English speaking or limited English proficiency
Older adults
Racially/ethnically diverse
Refugee
Substance dependent
Veterans
Other: ________________________
What types of services do you provide that are designed to impact climate change? Please select all of the services that your organization provides and indicate how long you have been providing the service (e.g., 2 years, 6 months).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For services connected to climate change that were selected in item 9, please describe (in greater detail) the activities that are being implemented to address climate change? ___________________
__________________________________________________________________________________
__________________________________________________________________________________
How has the COVID-19 pandemic impacted the services and activities provided by your organization? ______________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Partnerships
Does your organization partner with other organizations to provide activities or services for this program?
Yes
No
If yes is selected, go to item 13. If no is selected, go to item 22.
Please specify the number of organizations that your organization works with to provide activities or services for this program: ____________
If you worked with more than one partner, please think about one partnership that was successful and another partnership that was challenging when responding to the following questions. You will have the opportunity to answer each question twice to provide information about each partnership.
(Questions 14-21 will be shown twice so the grantee/sponsor can provide information about two organizations; they will have the option to skip the second series of questions if they have only one partner example.)
Partner organization name: ____________________________________________________
What types of activities or services does your partner organization provide for this program? (Select all that apply.)
Recruiting national service members or volunteers
Curriculum design for training
Identifying knowledge and skills for job requirements
Training national service members or volunteers to develop required skills
Providing facilities or equipment for training activities
Providing equipment or software to support program success
Coordinating program activities
Supporting national service members or volunteers while on the job
Other (please specify): ________________________
Prior to the current program, how often did your organization work with your partner organization?
Never
Rarely
Occasionally
Frequently
Always
Don’t know
Overall, how satisfied are you with the relationship with your partner?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
How would you classify your organization’s relationship with your partner organization?
We have a true partnership; we share strategies, resources, and information and jointly make decisions about the program.
We coordinate with the organization and make some of our decisions together about the program but the roles and responsibilities are clearly defined.
We cooperate with the organization to implement the program; we consider this organization when making decisions, but do not make decisions together.
We have limited communication; the organization does not influence our decisions and we do not influence their decisions about the program.
How would you describe your relationship with this partner organization?
Challenging
Successful
Please elaborate on the description that you gave to the relationship with your partner. Why was the relationship challenging or successful? ______________________________________________
_________________________________________________________________________________
The partnership between my organization and our partners will continue beyond the life of the grant period. (Using the selections below, please rate how strongly you agree with this statement.)
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
Program Outcomes
The items in this section are designed to measure your perceptions of the effectiveness of [program name].
Please indicate the level to which you think the program is effective at addressing the following in the community. If you do not have enough information to respond to the item, please select “Don’t know.”
|
Not Effective |
Somewhat Effective |
Effective |
Very Effective |
Don’t Know |
Increase the use of renewable energy sources |
|
|
|
|
|
Improve energy efficiency |
|
|
|
|
|
Prepare for extreme weather and climate |
|
|
|
|
|
Prepare for, respond to, or recover from natural disasters |
|
|
|
|
|
Preserve public lands and/or waterways |
|
|
|
|
|
Protect and/or restore biodiversity |
|
|
|
|
|
Help the community become more resilient to a changing climate |
|
|
|
|
|
Increase community engagement in climate change activities |
|
|
|
|
|
Increase self-sufficiency among community members |
|
|
|
|
|
Focus on the needs of vulnerable populations (economic, health, housing, transportation, social) |
|
|
|
|
|
Increase community members’ awareness of resources to meet their basic needs (e.g., food, shelter, medical) |
|
|
|
|
|
Increase community members’ knowledge about climate change |
|
|
|
|
|
Increase community members’ positive attitudes toward activities that mitigate climate change |
|
|
|
|
|
Increase community members’ behaviors around climate change |
|
|
|
|
|
Recruit diverse program participants |
|
|
|
|
|
Support local workforce development needs |
|
|
|
|
|
Increase skills development toward future employment |
|
|
|
|
|
Increase skills development toward green jobs |
|
|
|
|
|
Please indicate the level to which you think the program is effective at addressing the following for national service members. If you do not have enough information to respond to the item, please select “Don’t know.”
|
Not Effective |
Somewhat Effective |
Effective |
Very Effective |
Don’t Know |
Attitudes and behavior toward civic engagement |
|
|
|
|
|
Future career goals |
|
|
|
|
|
Future education goals |
|
|
|
|
|
Job skills development |
|
|
|
|
|
Skills development toward green jobs |
|
|
|
|
|
Knowledge of the impacts of climate change |
|
|
|
|
|
Activities that reduce greenhouse gas emissions |
|
|
|
|
|
Behavior to become more resilient to a changing climate |
|
|
|
|
|
Behavior to prepare for extreme weather and climate |
|
|
|
|
|
Behavior to become more resilient to extreme events |
|
|
|
|
|
In your opinion, what is contributing to achieving the desired outcomes in your community? ________________________________________________________________________________
What are the barriers to achieving the intended outcomes in your community? ________________ _________________________________________________________________________________
What recommendations do you have for other organizations that are planning to implement a similar program? __________________________________________________________________ _________________________________________________________________________________
Do you have any other comments? ___________________________________________________
Partner Organization Survey Consent
Participation
Thank you for taking the time to participate in this survey. This survey is part of a study being conducted by ICF to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your organization’s activities, partnership with [program name], and effect on the community you serve. The survey will take about 20 minutes to complete.
Risks
We do not anticipate any risks in participating in this survey. Participation in this survey is voluntary. You can skip any question or stop answering questions at any time.
Your responses to the survey will be kept confidential. Your answers will not affect your current or future work with [program name] or AmeriCorps. A report summarizing feedback across different organizations and AmeriCorps members will be shared with AmeriCorps. The information that we use from this survey will not be identified with any one individual. Only ICF staff and the AmeriCorps Office of Research and Evaluation will have access to the de-identified survey data.
Questions
Should you have any questions about the survey, you may contact study representative Dr. Kimberly Good at [email protected] or Institutional Review Board representative [name].
If you agree to participate in the survey, please acknowledge below by selecting, “I agree to participate.”
I agree to participate.
I do not agree to participate. [If selected, will use skip logic take respondent to the end of the survey]
Note: ICF will ask the grantee/sponsor to select 1–2 key partners to complete this survey.
Background
What is your current role at [organization name]?
Organizational leadership (e.g., executive director, president, CEO)
Office or program director
Office or program staff
Administrative staff (e.g., finance, development, human resources)
Volunteer
Other (please specify): ____________
How long have you been with [organization name]?
Less than 1 year
1–5 years
6–10 years
11–15 years
16+ years
How long have you been with [organization name] in your current role?
Less than 1 year
1–5 years
6–10 years
11–15 years
16+ years
What is your organization’s sector? (Select all that apply.)
Private, for-profit business
Public / government / Tribal organization, office, or agency
Nonprofit organization
Religious or other spiritual organization
School (K–12)
College or university
Other: ________________________
In general, what types of services does your organization provide? (Select all that apply.)
Education (e.g., tutoring, literacy, education for children and youth)
Health (e.g., nutrition, access to care, prevention, awareness, mental health)
Services to veterans, members of the armed forces, or their families
Environment (e.g., education on environmental issues, land conservation, energy conservation, ecosystem development/maintenance)
Disaster services (e.g., disaster education, preparation, mitigation, response)
Economic opportunity (e.g., housing services, employment counseling, job training, financial literacy, financial assistance)
Other
services (please specify): ____________________________________
Who does your organization serve? Please select the populations that your organization serves. (Select all that apply.)
Formerly Incarcerated
Homeless
Immigrant (documented and undocumented)
Individuals with disabilities
Low-income
Non-English speaking or limited English proficiency
Older adults
Racially/ethnically diverse
Refugee
Substance dependent
Veterans
Other:
________________________
What types of services does your organization provide that are designed to impact climate change? Please select all of the services that your organization provides and indicate how long you have been providing the service (e.g., 2 years, 6 months).
|
Service |
Length |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For services connected to climate change that were selected in item 7, please describe (in greater detail) the activities that are being implemented to address climate change?
______________________________________________________________________________
______________________________________________________________________________
How has the COVID-19 pandemic impacted the services and activities provided by your organization? ______________________________________________________________________________
______________________________________________________________________________
Partner Services
What types of activities or services does your organization provide for this program? (Select all that apply.)
Recruiting national service members or volunteers
Curriculum design for training
Identifying knowledge and skills for job requirements
Training national service members or volunteers to develop required skills
Providing facilities or equipment for training activities
Providing equipment or software to support program success
Coordinating program activities
Supporting national service members or volunteers while on the job
Other (please specify): ________________________
Prior to the current program, how often did your organization work with [grantee organization name]?
Never
Rarely
Occasionally
Frequently
Always
Don’t know
Overall, how satisfied are you with the relationship with [grantee organization name]?
Very satisfied
Satisfied
Neither satisfied nor dissatisfied
Dissatisfied
Very dissatisfied
How would you classify your organization’s relationship with [grantee organization name]?
We have a true partnership; we share strategies, resources, and information and jointly make decisions about the program.
We coordinate with the organization and make some of our decisions together about the program but the roles and responsibilities are clearly defined.
We cooperate with the organization to implement the program; we consider this organization when making decisions but do not make decisions together.
We have limited communication; the organization does not influence our decisions and we do not influence their decisions about the program.
Please rate how strongly you agree with this statement: The partnership between my organization and [grantee organization name] will continue beyond the life of the grant period.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
Program Outcomes
The items in this section are designed to measure your perceptions of the effectiveness of [program name].
Please indicate the level to which you think the program is effective at addressing the following in the community. If you do not have enough information to respond to the item, please select “Don’t know.”
|
Not Effective |
Somewhat Effective |
Effective |
Very Effective |
Don’t Know |
Increase the use of renewable energy sources |
|
|
|
|
|
Improve energy efficiency |
|
|
|
|
|
Prepare for extreme weather and climate |
|
|
|
|
|
Prepare for, respond to, or recover from natural disasters |
|
|
|
|
|
Preserve public lands and/or waterways |
|
|
|
|
|
Protect and/or restore biodiversity |
|
|
|
|
|
Helps the community become more resilient to a changing climate |
|
|
|
|
|
Increase community engagement in climate change activities |
|
|
|
|
|
Increase self-sufficiency among community members |
|
|
|
|
|
Focus on the needs of vulnerable populations (economic, health, housing, transportation, social) |
|
|
|
|
|
Increase community members’ awareness of resources to meet their basic needs (e.g., food, shelter, medical) |
|
|
|
|
|
Increase community members’ knowledge about climate change |
|
|
|
|
|
Increase community members’ positive attitudes toward activities that mitigate climate change |
|
|
|
|
|
Increase community members’ behaviors around climate change |
|
|
|
|
|
Recruit diverse program participants |
|
|
|
|
|
Support local workforce development needs |
|
|
|
|
|
Increase skills development toward future employment |
|
|
|
|
|
Increase skills development toward green jobs |
|
|
|
|
|
In your opinion, what is contributing to achieving the desired outcomes in your community? ____________________________________________________________________________________________________________________________________________________________________
What are the barriers to achieving the intended outcomes in your community? ____________________________________________________________________________________________________________________________________________________________________
What recommendations do you have for other organizations who are planning to implement a similar program? __________________________________________________________________________________ __________________________________________________________________________________
Do you have any other comments? ____________________________________________________________________________________________________________________________________________________________________
(Research question addressed is in parentheses)
Grantee/Sponsor Project Director Interview Consent
Participation
Thank you again for taking the time to participate in today’s interview. This interview is part of a study being conducted by ICF for AmeriCorps. The reason for the study is to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your program/project’s activities and effect on the community you serve. We have prepared a number of discussion questions, but please feel free to offer any additional thoughts you have about your experience. The interview will take up to 60 minutes.
Risks
We do not anticipate any risks in being a part of this interview. Participation in today’s discussion is completely up to you. You can decide you don’t want to answer any question we ask, and you can stop participating in the discussion at any time. You can also decide not to participate at all and leave the interview at any point.
Anything you share with ICF will be kept confidential. By speaking with us, you will not affect your current or future relationship with AmeriCorps. A report summarizing feedback across different organizations and AmeriCorps members will be shared with AmeriCorps. Only ICF staff will see the data collected from this interview, and the information that we use from our discussions will not be identified with any one individual. However, since we are only talking to a small number of grantee staff at each organization, there is a chance that AmeriCorps personnel will be able to guess which individual shared certain information.
Do you have any questions about this study or this interview?
Before we begin, we would also like to check that you are 18 years or older and get your permission to participate in today’s discussion. If you do not want to participate, you may leave at this time.
Consent Statement
You are 18 years or older.
You understand being a part of this study is completely up to you and that you can stop being a part of the discussion at any time, with no penalty or risk.
You understand that only ICF staff will see your answers to these questions.
You understand that your name will not be included in any reports or presentations of the results and that what you share with us today will be treated as confidential.
You understand the possible risks and benefits of being a part of this study.
Questions
Should you have any questions about the interview, you may contact study representative Dr. Kimberly Good at [email protected], or Institutional Review Board representative [name].
Background
Let’s begin with some brief introductions. Please tell me a little bit about yourself.
What is your title and role at [organization name]?
How long have you been in this position?
What are your specific responsibilities?
Can you tell me a little about [organization name]? What types of services and activities does your organization provide?
Were you required to provide matching funds to receive AmeriCorps funding for the [program name]? If yes, to what extent were the required matching funds easily secured? Explain.
Program Operations and Context
I’d like to ask you a few questions about your program, [program name], and the community that you serve.
What are the mission and goals of [program name]? (1)
What are program priorities? (1.a)
In what ways has climate change affected your community and its members? (8)
Probe for economic, health, housing, transportation, and social impacts
Can you discuss some of the services and activities that address climate change issues in your community? (1.b)
How are you measuring program impacts? (1.c)
Use the probes below based on the specific program
Capacity
Building
Number of organizations that received
capacity building services/increased their effectiveness,
efficiency, and/or program scale/reach
Disaster
Services
Number of individuals served; number of
individuals reporting increased disaster readiness; number of
structures protected or restored/returned to regular use after a
disaster
Environmental
Stewardship
Number of housing units or public
structures weatherized or retrofitted to improve energy
efficiency/with reduced energy consumption or reduced energy costs;
number of individuals receiving education or training in
environmental stewardship and/or environmentally-conscious
practices/with increased knowledge of environmental stewardship
and/or environmentally-conscious practices; number reporting a
change in behavior or the intent to change behavior to better
protect the environment; number of acres of public parks or other
public and Tribal lands that are treated/improved; number of miles
of public trails or waterways that are treated and/or
constructed/improved and/or put into use
What are some challenges that you’ve encountered with the AmeriCorps performance measures? This includes outputs and outcomes.
How does your program define equity? (2)
Does the program have goals around equity and inclusion? If yes, please tell us about them and how they are measured.
Can you describe the strategies that your program uses to increase equity?
What are the characteristics of the communities and individuals served through the program? (9.a)
Can you speak to the role of the community in this program?
How were community members notified about the program?
How are community members involved in program activities? (10.a) Are they actively involved in program activities? What role do they have in the program?
Were the community members involved in program planning? If yes, please tell us about the process.
Thinking about how this program was structured and implemented, is the program operating as intended? (3)
How did the COVID-19 pandemic affect program planning and implementation? (12)
How did your organization adapt during the pandemic? In what ways are you adapting program activities and services? (12.b)
Was the training of national service members adapted? If so, how? (12.a)
What adaptations will you retain moving forward? Why? (12.c)
What aspects of the program do you think are most effective? What do you think could be improved in how the program was structured and implemented? (4)
What practices have been most effective for engaging vulnerable and/or diverse communities? (4.a) Is there anything that you would do differently to increase community engagement?
What practices have been most effective for increasing equity? What gaps still need to be addressed?
What barriers has the program faced while trying to meet intended outcomes? (5)
What facilitators have assisted the program in meeting intended outcomes? (5)
Considering the goal of addressing climate change, what successful strategies or innovations did your program implement that could be transferable to other communities or organizations wanting to implement this type of program? (6)
Partnerships
What are the number and types of partners that your organization is working with on [program name]?
Do you have any partnerships that focus on resilience building for at-risk populations within your community? (Examples may include racial minority groups, populations with limited English proficiency, disabled populations, and any other identified populations within your community.) (14)
Can you please tell me how you started working with [partner organization name]? How long have you been working with them? (16)
How has [organization name] maintained a relationship with [partner organization name] during that time? (16)
How often do you engage with your partners? (14)
How often do you work with your partners on joint activities?
How would you describe the relationship between the organizations? (15)
How is your organization working with [partner organization name] to build community resilience? (16)
Program Outcomes
In what ways does your program improve member outcomes? (19 & 21)
Probe for increased civic engagement (19.a)
Probe for increased knowledge of the impacts and solutions to climate change, positive attitudes toward activities that mitigate climate change, and behaviors around climate change
Probe for an increase in functional and technical job skills, interest/willingness to pursue a career in a green job (21.a, 21.b)
From your perspective, in what ways has [program name] helped the community address climate change?
Probe on the following and follow up with how
Energy efficiency/renewable energy sources (18.a)
Disaster response (18.b)
Environmental conservation (18.c)
Improved knowledge, attitudes, and behaviors around climate change (18.d)
Built community capacity to mitigate and adapt to climate change (18.e)
In what ways has your program increased equity in the community?
Probes include increased education, engagement, self-sufficiency, and partnerships
Program Sustainability
From your perspective, how likely is it that you will be able to sustain the program after the grant-funding period? (7)
Which components will you be able to sustain?
Do you have any additional feedback or insights you would like to share with us regarding the program?
GRANTEE/SPONSOR
ORGANIZATION –
PROGRAM STAFF IN-DEPTH INTERVIEW
(60
minutes)
Grantee Staff Interview Consent
Participation
Thank you again for taking the time to participate in today’s interview. This interview is part of a study being conducted by ICF for AmeriCorps. The reason for the study is to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your program’s activities and effect on the community you serve. We have prepared a number of discussion questions, but please feel free to offer any additional thoughts you have about your experience. The interview will take up to 60 minutes.
Risks
We do not anticipate any risks in being a part of this interview. Participation in today’s discussion is completely up to you. You can decide you don’t want to answer any question we ask you, and you can stop participating in the discussion at any time. You can also decide not to participate at all and leave the interview at any point.
Anything you share with ICF will be kept confidential. By speaking with us, you will not affect your current or future relationship with AmeriCorps. A report summarizing feedback across different organizations and AmeriCorps members will be shared with AmeriCorps. Only ICF staff will see the data collected from this interview, and the information that we use from our discussions will not be identified with any one individual. However, since we are only talking to a small number of grantee staff at each organization, there is a chance that AmeriCorps personnel will be able to guess which individual shared certain information.
Do you have any questions about this study or this interview?
Before we begin, we would also like to check that you are 18 years or older and get your permission to participate in today’s discussion. If you do not want to participate, you may leave at this time.
Consent Statement
You are 18 years or older.
You understand being a part of this study is completely up to you and that you can stop being a part of the discussion at any time, with no penalty or risk.
You understand that only ICF staff will see your answers to these questions.
You understand that your name will not be included in any reports or presentations of the results and that what you share with us today will be treated as confidential.
You understand the possible risks and benefits of being a part of this study.
Questions
Should you have any questions about the interview, you may contact study representative Dr. Kimberly Good at [email protected], or Institutional Review Board representative [name].
Background
Let’s begin with some brief introductions. Please tell me a little bit about yourself.
Program Operations and Context
How have you been involved with [program name]?
In what ways has climate change affected your community and members of the community? (8)
Probe for economic, health, housing, transportation, and social impacts
Can you discuss some of the services and activities that address climate change? (1.b)
Can you discuss how the program attempts to increase equity in terms of climate change outcomes? (2)
Do you work with the national service members? How are the members recruited? Can you tell us about their activities and your interactions with the members?
Probe for information about the characteristics of the national service members
Do you work with members of the community? How are they involved in the program?
Probe for information about the characteristics of the community and its members; probe for community members’ role in program activities
What practices have you found to be most effective at engaging vulnerable and/or diverse communities? (4.a)
How did your organization adapt during the COVID-19 pandemic? (12)
What aspects of the program do you think are most effective? What do you think could be improved in how the program was structured and implemented? (4)
How are outcomes related to climate change measured? (1.c)
Use the probes below based on the specific program
Capacity
Building
Number of organizations that received
capacity building services/increased their effectiveness,
efficiency, and/or program scale/reach
Disaster
Services
Number of individuals served; number of
individuals reporting increased disaster readiness; number of
structures protected or restored/returned to regular use after a
disaster
Environmental
Stewardship
Number of housing units or public
structures weatherized or retrofitted to improve energy
efficiency/with reduced energy consumption or reduced energy costs;
number of individuals receiving education or training in
environmental stewardship and/or environmentally-conscious
practices/with increased knowledge of environmental stewardship
and/or environmentally-conscious practices; number reporting a
change in behavior or the intent to change behavior to better
protect the environment; number of acres of public parks or other
public and Tribal lands that are treated/improved; number of miles
of public trails or waterways that are treated and/or
constructed/improved and/or put into use
What are some challenges that you’ve encountered with the AmeriCorps performance measures? This includes outputs and outcomes.
From your perspective, what challenges did your organization encounter in meeting intended outcomes? (5)
What were the most successful strategies that you observed to meet intended outcomes? (5)
Considering the goal of addressing climate change, what successful strategies or innovations did your program implement that could be transferable to other communities or organizations wanting to implement this type of program? (6)
Working with National Service Members
Does [organization name] offer members any additional training? If so, what types of training? (20, 21.a)
Probe for member skill development toward green jobs and willingness to pursue a career in conservation and environmental education fields (21.a, 21.b)
What have you found to be best practices for engaging members? What are the challenges? (4)
Do you have any specific practices to effectively support members from diverse backgrounds? (4.a)
Probe for diversity, equity, and inclusion (DE&I) commitments, affinity groups, identity-specific outreach events, and so forth
Working with Partner Organizations
Does [organization name] work with community partners? If so, can you please share what type of collaboration you have had with these partners in your work on [program name]? (14, 15)
How often do you engage with your partners? (15)
How often do you work with these partners on joint activities?
How has [organization name] built and maintained these partnerships? (16)
Do you think your work with partners helps build community resilience? If so, how? (17)
Program Outcomes
From your perspective, in what ways has [program name] helped the community address climate change?
Probe on the following and follow up with how
Energy efficiency/renewable energy sources (18.a)
Disaster response (18.b)
Environmental conservation (18.c)
Improved knowledge, attitudes, and behaviors around climate change (18.d)
Built
community capacity to mitigate and adapt to climate change
(18.e)
In what ways has your program increased equity in the community?
Probes include increased education, engagement, self-sufficiency, and partnerships
In what ways has [program name] improved member outcomes? (19 & 21)
Probe for increased civic engagement (19.a)
Probe for members’ increase in functional and technical job skills, interest/willingness to pursue a career in a green job (21.a, 21.b)
Probe for previous members’ job attainment and careers in green jobs after their service (21.c, 21.d)
Do you have any additional feedback or insights you would like to share with us regarding the program?
AMERICORPS
MEMBER FOCUS GROUP
(90 minutes)
AmeriCorps Member Focus Group Consent
Participation
Thank you again for taking the time to participate in today’s focus group. This focus group is part of a study being conducted by ICF for AmeriCorps. The reason for the study is to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your program’s activities and effect on the community you serve. We have prepared a number of discussion questions, but please feel free to offer any additional thoughts you have about your experience. The focus group will take up to 60 minutes.
Risks
We do not anticipate any risks in being a part of this focus group. Participation in today’s discussion is completely up to you. You can decide you don’t want to answer any question we ask you, and you can stop participating in the discussion at any time. You can also decide not to participate at all and leave the focus group at any point.
Anything you share with ICF will be kept confidential. By speaking with us, you will not affect your current or future relationship with [program name]. A report summarizing feedback across different organizations and AmeriCorps Members will be shared with AmeriCorps. Only ICF staff will see the data collected from this focus group, and the information that we use from our discussions will not be identified with any one individual. However, since we are only talking to a small number of members at each organization, there is a chance that AmeriCorps or [program name] personnel will be able to guess which individual shared certain information.
We also request that you do not discuss what is disclosed in this focus group once the discussion ends.
Do you have any questions about this study or this focus group?
Before we begin, we would also like to check that you are 18 years or older and get your permission to participate in today’s discussion. If you do not want to participate, you may leave at this time.
Consent Statement
You are 18 years or older.
You understand being a part of this study is completely up to you and that you can stop being a part of the discussion at any time, with no penalty or risk.
You understand that only ICF staff will see your answers to these questions.
You understand that your name will not be included in any reports or presentations of the results and that what you share with us today will be treated as confidential.
You understand the possible risks and benefits of being a part of this study.
Questions
Should you have any questions about the focus group, you may contact study representative Dr. Kimberly Good at [email protected], or Institutional Review Board representative [name].
Let’s begin with some brief introductions. Please tell us your name and a little bit about yourself.
Background
Why did you become a national service member?
To what extent were you engaged in community activities prior to your service?
Why did you choose to serve in [program name]? How did you become aware of the program?
Are you a member of the community that you are serving?
How did the COVID-19 pandemic affect you and your community? (13)
Program Context and Activities
How does climate change impact your life? In what ways has climate change affected your community and members of the community? (8)
Probe for economic, health, housing, transportation, and social impacts
What is your understanding of the mission and goals of [program name]? In your own words, what is the program designed to accomplish?
What activities and services are intended to address climate change? (1.b)
Can you discuss how the program attempts to increase equity in terms of climate change outcomes? (2)
How have you been involved with [program name]?
What is your role? What services did you provide? What activities did you engage in during your service?
Do you work with members of the community? How are they involved in the program?
Probe for information about the characteristics of the community and its members; probe for community members’ role in program activities (10)
How did the COVID-19 pandemic affect program activities and services? (12)
Training and Supports
Overall, how prepared were you for your position as a national service member?
Now please think about your specific job tasks—for example, conducting community outreach or retrofitting a building. How prepared did you feel to perform these tasks?
After the training, how knowledgeable were you about the impacts of and solutions to climate change?
Were you provided a living allowance and/or stipend for participating in the program? If yes, were these monetary supports sufficient to meet your living needs? Why or why not?
Program Outcomes
In what ways has [program name] affected you personally? (19 & 21)
Probe for increased civic engagement (19.a)
Probe for increased knowledge, attitudes, and behaviors around climate change
Probe for an increase in functional and technical job skills, interest/willingness to pursue a career in a green job (19.a, 19.b)
Following your experience in the program, would you be interested in pursuing additional opportunities to positively engage with your community? (19)
Examples may include volunteer work, contributions to charitable organizations within your community, participation in discussions concerning social/environmental responsibility, and so forth.
Do you have any interest in obtaining a job related to climate change? (21.b)
If yes, probe to find out what area (e.g., energy, conservation)
If yes, probe to find out if they will have need to pursue more schooling and in what field of study
Can you tell me about how your experience has affected your preparedness for a career within the climate change field? Have you developed necessary skills for a career within climate change? (20, 21.a)
If not, has your experience within the program increased your awareness of pathways to obtain necessary skills for a career within climate change?
From your perspective, in what ways has [program name] helped the community address climate change?
Probe on the following and follow up with how
Energy efficiency/renewable energy sources (18.a)
Disaster response (18.b)
Environmental conservation (18.c)
Improved knowledge, attitudes, and behaviors around climate change (18.d)
Built community capacity to mitigate and adapt to climate change (18.e)
Final Thoughts
Do you have any suggestions to improve [program name]? (6)
Probe for training/onboarding deficiencies, member-level challenges, etc.
Do you have any additional feedback or insights you would like to share with us regarding your experiences in the program or working with [organization name]?
COMMUNITY
MEMBER FOCUS GROUP
(60 minutes)
Community Member Focus Group Consent
Participation
Thank you again for taking the time to participate in today’s focus group. This focus group is part of a study being conducted by ICF for AmeriCorps. The reason for the study is to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your program’s activities and effect on the community you serve. We have prepared a number of discussion questions, but please feel free to offer any additional thoughts you have about your experience. The focus group will take up to 60 minutes.
Risks
We do not anticipate any risks in being a part of this focus group. Participation in today’s discussion is completely up to you. You can decide you don’t want to answer any question we ask you, and you can stop participating in the discussion at any time. You can also decide not to participate at all and leave the focus group at any point.
Anything you share with ICF will be kept confidential. By speaking with us, you will not affect your current or future relationship with [program name]. A report summarizing feedback across different organizations and AmeriCorps Members will be shared with AmeriCorps. Only ICF staff will see the data collected from this focus group, and the information that we use from our discussions will not be identified with any one individual. However, since we are only talking to a small number of members at each organization, there is a chance that AmeriCorps or [program name] personnel will be able to guess which individual shared certain information.
We also request that you do not discuss what is disclosed in this focus group once the discussion ends.
Do you have any questions about this study or this focus group?
Before we begin, we would also like to check that you are 18 years or older and get your permission to participate in today’s discussion. If you do not want to participate, you may leave at this time.
Consent Statement
You are 18 years or older.
You understand being a part of this study is completely up to you and that you can stop being a part of the discussion at any time, with no penalty or risk.
You understand that only ICF staff will see your answers to these questions.
You understand that your name will not be included in any reports or presentations of the results and that what you share with us today will be treated as confidential.
You understand the possible risks and benefits of being a part of this study.
Questions
Should you have any questions about the focus group, you may contact study representative Dr. Kimberly Good at [email protected], or Institutional Review Board representative [name].
Let’s begin with some brief introductions. Please tell us your name and a little bit about yourself.
Participants introduce themselves
Community Context
Can you tell us about your community? What is it like? (9)
Probe for history, infrastructure, community leaders
How long have you lived in the community? Has it changed in the past 5 years? If so, how?
How did the COVID-19 pandemic affect you and your community?
Perceptions of Climate Change
What do you know about climate change?
Probe for causes of climate change, their beliefs about climate change
How does climate change impact your life? (8)
Probe for economic, health, housing, transportation, and social impacts
What are you doing to respond to climate change?
Engagement with the Program
Why did you get involved with [program name]?
How have you been involved with [program name]? (10.a)
Did you provide feedback on the program? Did you contribute to the development, content, or activities of the program?
Did you receive any training or education related to climate change? If so, what was the training about and how helpful was it?
Program Outcomes
In what ways has [program name] affected you personally?
Probe on the following and follow up with how
Energy efficiency/renewable energy sources (18.a)
Disaster response (18.b)
Environmental conservation (18.c)
Improved knowledge, attitudes, and behaviors around climate change (18.d)
Built community capacity to mitigate and adapt to climate change (18.e)
Do you have any suggestions to improve [program name]?
Partner Organization Consent Statement
Participation
Thank you again for taking the time to participate in today’s interview. This interview is part of a study being conducted by ICF for AmeriCorps. The reason for the study is to help AmeriCorps and [program name] better understand how their programs are working. More specifically, we are interested in your organization’s activities, partnership with [program name], and effect on the community you serve. We have prepared a number of discussion questions, but please feel free to offer any additional thoughts you have about your experience. The interview will take about 30 minutes.
Risks
We do not anticipate any risks in being a part of this interview. Participation in today’s discussion is completely up to you. You can decide you don’t want to answer any question we ask you, and you can stop participating in the discussion at any time. You can also decide not to participate at all and leave the interview at any point.
Anything you share with ICF will be kept confidential. By speaking with us, you will not affect your current or future relationship with [program name] or AmeriCorps. A report summarizing feedback across different organizations and AmeriCorps members will be shared with AmeriCorps. Only ICF staff will see the data collected from this interview, and the information that we use from our discussions will not be identified with any one individual.
Do you have any questions about this study or this interview?
Before we begin, we would also like to check that you are 18 years or older and get your permission to participate in today’s discussion. If you do not want to participate, you may leave at this time.
Consent Statement
You are 18 years or older.
You understand being a part of this study is completely up to you and that you can stop being a part of the discussion at any time, with no penalty or risk.
You understand that only ICF staff will see your answers to these questions.
You understand that your name will not be included in any reports or presentations of the results and that what you share with us today will be treated as confidential.
You understand the possible risks and benefits of being a part of this study.
Questions
Should you have any questions about the interview, you may contact study representative Dr. Kimberly Good at [email protected], or Institutional Review Board representative [name].
Background
Let’s begin with some brief introductions. Please tell me a little bit about yourself.
What is your title and role at [partner organization name]?
How long have you been in this position?
What are your specific responsibilities?
Can you tell me a little about [partner organization name]? What types of services and activities does your organization provide?
Partnership and Program Operations
Can you please tell me how you started working with [organization name]? How long have you been working with them? (16)
How has [partner organization name] maintained a relationship with [organization name] during that time? (16)
How would you describe the relationship between the organizations? (15)
What role does your organization play with [program name]? How have you been involved with planning/implementation of [program name]? (14)
How did the COVID-19 pandemic affect program planning and implementation? (12)
How is your organization building community resilience? (17)
In what ways has climate change affected your community and its members? (8)
Probe for economic, health, housing, transportation, and social impacts
Can you discuss some of the services and activities that address climate change?
Can you discuss how the program attempts to increase equity in terms of climate change outcomes? (2)
Do you work with members of the community? How are they involved in the program? (10.a)
Probe to identify if they were involved in planning and implementation
What practices have you found to be most effective for engaging vulnerable and/or diverse communities? (4.a)
What aspects of the program do you think are most effective? What do you think could be improved in how the program was structured and implemented? (4)
Program
Outcomes
From your perspective, in what ways has [program name] helped the community address climate change?
Probe on the following and follow up with how
Energy efficiency/renewable energy sources (18.a)
Disaster response (18.b)
Environmental conservation (18.c)
Improved knowledge, attitudes, and behaviors around climate change (18.d)
Built community capacity to mitigate and adapt to climate change (18.e)
From your perspective, how likely is it that you will be able to sustain the program after the grant-funding period? (7)
Which components will you be able to sustain?
Do you have any additional feedback or insights you would like to share with us regarding [program name] or working with [organization name]?
On behalf of AmeriCorps (formerly known as the Corporation for National and Community Service), ICF is providing evaluation capacity building workshops throughout one year to grantees and sites that are participating in the [bundle name] evaluation. We are inviting you to complete this 15-minute survey at the beginning and end of your participation.
Participation in this survey is voluntary, but we hope you will respond because your experience and perspective are extremely valuable to assess the success of these workshops.
1. Awareness/Attitudes/Motivation:
To what extent do you agree with the following statements?
(1 – Strongly Disagree, 2 – Disagree, 3 – Neither Agree nor Disagree, 4 – Agree, 5 – Strongly Agree)
An evaluation…
will help me understand my program.
will inform the decisions I make about my program.
will justify funding for my program.
will help to convince managers that changes are needed in my program.
is absolutely necessary to improve my program.
should involve program participants in the evaluation process.
will help improve services to target populations.
takes away resources that should be used to provide services.
is incorporated into my daily work.
report is something I review regularly.
can have findings that affect the decisions I make in my work.
I am motivated to…
learn about evaluation.
start evaluating my program.
support other staff to evaluate their program.
encourage others to buy in to evaluating our program.
To what
extent are the following factors important in your motivation to
engage in program evaluation?
(1 – Not at all, 2 –
To a small extent, 3 – To a moderate extent, 4 – To a
large extent, 5 – Always)
A need to meet accountability requirements
A desire to enlighten and support government policy making and planning
Changes in the organization (e.g., reorganization, new leadership, or vision)
A mandate from leadership to increase the learning function of evaluation
A perceived lack of internal evaluation knowledge and skills
A desire to seek new or increased funding
A desire to use evaluation to make program improvements
2. Skills/Knowledge/Behaviors:
To what extent do you agree with the following statements?
(1 – Not at all, 2 – To a small extent, 3 – To a moderate extent, 4 – To a large extent, 5 – Always)
I know how to…
recognize key concepts in evaluation (evidence, data, performance measurement, evaluation).
recognize the components of an evaluation plan.
describe what a “theory of change” is.
recognize how theory of change connects to a logic model.
recognize the attributes of a good evaluation question.
identify strategies to collect information from participants.
recognize features of process and outcome indicators of a program.
recognize who should collect data.
recognize when data should be collected.
identify common data analysis terms.
recognize how quantitative and qualitative analysis is performed.
recognize how to use the evaluation findings to improve a program.
read an evaluation report and recognize its basic components.
In the past year, I helped my organization to…
provide funding to carry out an evaluation or hire an external evaluator (as part of a grant or through other means).
provide training or technical assistance to conduct an evaluation (by your staff, consultants, or other means).
conduct evaluations of programs funded by your organization.
use evaluation findings to improve programs funded by your organization.
use evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization.
Evaluation Needs
Does your organization have any of the following?
Staff position(s) or group within your organization dedicated to evaluation
External evaluation partner(s), consultant(s), or organization(s) that provide your organization with evaluation services
Part of the organizations' budget dedicated to evaluation
Are you currently working with an evaluator for this program?
Yes
No
To what extent have you experienced the following barriers to conducting a program evaluation? (Select all that apply)
Not enough time
Not enough money
Not knowing how to conduct a program evaluation
Not having assistance with data collection
Not getting enough people to respond to surveys
Not knowing how to analyze data
Not having people to turn to for consultation and assistance
Not knowing what questions to ask
Not knowing how to write up results
Other factors (please specify):
If you were to attend a series of evaluation-related workshop, what would you like to learn most? (Select only one.)
How to conduct an evaluation within my organization
How to work with external evaluators
What other organizations do with their evaluations
Less than 10 minutes
11–20 minutes
21–30 minutes
Which of the following approaches would you prefer:
To have discussion-centered learning activities but with an expectation that you read evaluation learning materials and complete activities in advance.
To have presentation-centered learning activities with little or no expectation to review or prepare materials in advance.
4. About the Respondent
A. Which one of these groups would you say best represents your race?
White
Black or African American
American Indian or Alaska Native
Asian
Native Hawaiian or Pacific Islander
Multiracial or multi-ethnic (2+ races/ethnicities)
Other (please specify): ____________
Don’t know
Prefer not to say
Are you of Hispanic, Latino/a, or Spanish origin?
Yes
No
Don’t know
Prefer not to say
What is your gender?
Male
Female
Nonconforming
Prefer not to respond
Prefer to self-describe (please specify): __________________________________
How long have you worked with your organization?
Less than 1 year
2–5 years
6–10 years
11–20 years
21+ years
What is your position in the organization?
Leader (CEO, executive Director)
Other executive
Manager/Supervisor
Staff
Board member
Other (please specify): __________________________________
How long has your organization worked with AmeriCorps?
Less than 1 year
2–5 years
6–10 years
11–20 years
21+ years
Module 1
Presentation 1: Introduction and Evaluation Basics
Learning Objectives
Describe how evaluations can be useful
Recognize key concepts in evaluation
Recognize stages of evaluation
Knowledge Questions
What is not a good reason to conduct a program evaluation?
Program A wants to demonstrate the benefits of the program to stakeholders
Program C wants to understand what needs to be improved in the program
Program B wants to generate a daily report for its employees
Program D wants to make decisions about resource allocation
Please match the following terms with the right definition.
Evidence: The reasons why you think your program will achieve its goals.
Evaluation: Systematic investigation of your program.
Quantitative data: Numerical information that can be counted, quantified, and mathematically analyzed.
Qualitative data: Narrative information that often describes context, motivations, and “How?” and “Why?” answers.
Which of these statements best highlights the difference between program evaluation and performance measurement?
Performance measurement uses a systematic approach while evaluation uses regular updates of changes over time.
Performance measurement answers program-level research questions while evaluation answers routine questions.
Performance measurement reports data that requires programmatic change while evaluation reports data that requires immediate change.
Performance measurement indicates whether a project is moving according to plan while evaluation systematically examines the workings of a program.
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much do you agree or disagree with the following statements. (Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
Why evaluations can be useful
The difference between evidence and data
The difference between evaluation and performance measurement
Presentation 2: Getting to Know One Another
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements? (Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
Getting to know my peers was useful to me.
The instructor(s) facilitated the session well.
Module 1
Presentation 3: Theory of Change
Knowledge Questions
Which of these best defines what a “theory of change” is?
An explanation of any negative or positive impact your program has on the community
An explanation of what your program’s ultimate goal is
An example of how you’d like to change your role within your program
A reflection on how you achieve your personal goals every day within your program
What is not a reason to develop a theory of change?
It ensures people will not ask you to evaluate your program
It forces your organization to articulate the assumptions underlying your program
It highlights the goals your program is trying to achieve
It shows the unique contribution your program makes to achieve its goals
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
What a theory of change is
How to come up with a theory of change for my program
Module 1
Presentation 4: Logic Models
Knowledge Questions
Select the statement that best describes how a logic model connects to your theory of change.
A theory of change visually describes a logic model.
A logic model visually shows how your program’s theory of change works.
A theory of change visually represents the connections between program components, while a logic model is a statement describing the program’s goals.
A logic model presents every theory of change that you can come up with in one chart.
Using what you’ve learned in class about what goes into a logic model and the following theory of change, select one item for each column in the logic model come up with one item for each column in the logic model.
Our state increasingly needs public lands maintenance. We believe this is because our state is facing an increase in natural and human-made disasters and does not have enough high-impact conservation efforts in place. Our theory is that if AmeriCorps members assist with high-impact conservation, disaster recovery, and mitigation projects, and reroute trails to avoid sensitive ecosystems, then public lands will become more resilient and resistant to future disasters as well as recover from past disasters. By engaging in high-impact conservation and treating various ecosystems, we will ease the burden of the disasters on our state. Ultimately, this will reduce the need for reactive public land maintenance.
An example of an input for this program:
An example of activities:
Funding for the program
Rerouting trails to avoid sensitive ecosystems
Number of trails that are constructed because of the program
Changes in invasive species numbers in parks
Reduced need for reactive public land maintenance
An example of outputs:
Funding for the program
Rerouting trails to avoid sensitive ecosystems
Number of trails that are constructed because of the program
Changes in invasive species numbers in parks
Reduced need for reactive public land maintenance
An example of short-term outcomes:
Funding for the program
Rerouting trails to avoid sensitive ecosystems
Number of trails that are constructed because of the program
Changes in invasive species numbers in parks
Reduced need for reactive public land maintenance
An example of intermediate outcomes:
Funding for the program
Rerouting trails to avoid sensitive ecosystems
Number of ecosystems that have recovered from disasters
Changes in invasive species numbers in parks
Reduced need for reactive public land maintenance
An example of long-term outcomes:
Funding for the program
Rerouting trails to avoid sensitive ecosystems
Number of trails that are constructed because of the program
Changes in invasive species numbers in parks
Reduced need for reactive public land maintenance
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
What a logic model is
What the difference is between a logic model and a theory of change
How to figure out what goes into a logic model
Module 1
Presentation 5: Evaluation Planning
Knowledge Questions
Which is an example of a good evaluation question?
Does the climate change program work?
Are the disaster mitigation strategies useful?
To what extent does the likelihood of public lands maintenance differ for communities who participate in this program compared to other communities?
Does the program improve its communities’ happiness?
What’s the difference between process and outcome evaluations?
Process evaluations examine how a program works while outcome evaluations measure the results or effects of a program
Process evaluations measure changes in attitudes while outcome evaluations examine whether the program has sufficient resources
Process evaluations measure changes to beneficiaries while outcome evaluations measure what pieces of the program are working as expected
Process evaluations measure the results of a program while outcome evaluations measure program activities
What is an example of a quantitative indicator for a process evaluation?
Community leaders’ thoughts about what did or did not work well in their program
AmeriCorps members’ opinions about what they liked and did not like about their training
The number of communities who received disaster relief initiatives
Attributes of communities who received all program components and those who did not
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
The difference between a process and an outcome evaluation
The attributes of a good evaluation question
Module 2
Presentation 1: Preparing to Collect Data
Knowledge Questions
What is an example of primary data that can be collected to evaluate the impact of the recovery coaching program?
Economic data on communities from the U.S. Census Bureau
Geologic basin boundaries from the U.S. Environmental Protection Agency
Survey responses from AmeriCorps members in the program
Community actions data from the state
When collecting data from the population, what is a good reason to draw a sample? (Select all that apply.)
It can be time-consuming to collect data from an entire population
It costs less to collect data from a sample than a population
One can get a higher response rate using follow-up efforts with data collected from a sample than a population
A sample can be representative of a population
When should you collect the following types of data?
(Before the intervention / During the intervention / After the intervention)
Baseline data: __________ (Before the intervention)
Number of rivers that were treated: ___________ (During or after the intervention)
Changes in employment: ___________ (After the intervention)
Satisfaction Questions
Overall, how satisfied are you with this module?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
The difference between primary data and secondary data
When data should be collected to evaluate a program
Module 2
Presentation 2: Data Collection Techniques
Knowledge Questions
If you are intending to collect qualitative data from a participant, which data collection technique would you not utilize?
Structured interviews
Focus groups
Structured observations
Closed-ended survey questions
Read the questions below and identify the “common pitfall” (if any) associated with each of the survey questions below.
Do you feel that you have had a great experience with treating public lands in this program?
Double-barreled question
Leading question
Social desirability
Non-applicable question
No pitfall
Please rate the impact of removing hazard trees to mitigate wildland fire, with 1 being “very useful” and 5 being “not useful at all.”
Double-barreled question
Leading question
Social desirability
Non-applicable question
No pitfall
Do you have any recommendations or complaints about the river treatment process?
Double-barreled question
Leading question
Social desirability
Non-applicable question
No pitfall
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
When to use surveys for primary data collection
When to use focus groups for primary data collection
When to use interviews for primary data collection
When to use structured observations for primary data collection
The different types of data collection techniques
Module 2
Presentation 3: Data Analysis
Knowledge Questions
What kinds of questions would quantitative data be used to answer?
How do AmeriCorps members interact with the organization when assisting with conservation efforts?
Why do some communities become resilient while others do not?
How did clearing trails affect erosion and flooding on public lands?
How many miles of trails have been created by AmeriCorps members?
What is an example of qualitative analysis of data?
Frequencies
Counts
Minimum and maximum
Interpretation of direct quotes
What is an effective way to quantitively analyze data?
Measure whether a difference between two groups is “statistically significant”
Code participant responses to analyze themes in the data
Interpret differences in categories of coded responses
Taking narrative notes
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
Quantitative analysis techniques
Qualitative analysis techniques
Module 3
Presentation 1: Evaluation Reporting
Knowledge Questions
Which of these questions should the discussion section of an evaluation report answer?
What was the approach used to analyze the data?
What was the theory of change for the program?
Do the findings of the report support your theory of change?
Which of these questions should the methodology section of an evaluation report answer?
What was the approach used to analyze the data?
What was the theory of change for the program?
Do the findings of the report support your theory of change?
Which of these questions should the introduction section of an evaluation report answer?
What was the approach used to analyze the data?
What was the theory of change for the program?
Do the findings of the report support your theory of change?
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
What goes into an evaluation report
Module 3
Presentation
2: Using Evaluation for Program Improvement
Knowledge Questions
Select all the ways you can use findings from a process evaluation for program improvement.
To demonstrate improvements for communities after disaster recovery initiatives
To plan further evaluations to examine long-term reductions in community resilience
To demonstrate changes in land erosion after conservation efforts through the program
To change the duration of maintenance the public lands receive
To revise the intervention model
Select all the ways you can use findings from an outcome evaluation for program improvement.
To demonstrate improvements for communities after disaster recovery initiatives
To plan further evaluations to examine long-term reductions in community resilience
To demonstrate changes in land erosion after conservation efforts through the program
To change the duration of maintenance the public lands receive
To revise the intervention model
To change the program activities
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
Findings from a process evaluation
Findings from an outcome evaluation
Module 3
Presentation 3: Developing a Long-Term Research Agenda
Knowledge Questions
What does not characterize a long-term research agenda?
It should be progressive
Once it is established, it cannot be changed
It should build on what has been done in the past
It should point towards future goals
What needs to be done to build evidence to support the program goals over time? Arrange the following tasks in order of complexity.
Build consensus about your overall evaluation goals.
Conduct a process evaluation.
Conduct an outcome evaluation.
Conduct an impact evaluation with a comparison group.
Expand the program to multiple sites.
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
Understand the key components of a long-term research agenda.
Visualize how to develop a long-term research agenda for your program.
Module 3
Presentation
4: Application to Climate Change Bundle
Satisfaction Questions
Overall, how satisfied are you with this presentation?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree)
The information I learned in this module was useful for me.
The instructor(s) clearly explained the material covered in this module.
I felt comfortable voicing any questions I had in class.
The activities were helpful to my understanding of the material.
I understand how to apply this material to my own program.
After taking this module, how do you rate your understanding of the following topics?
(Very Poor, Poor, Acceptable, Good, Very Good)
How the climate change bundle follows evaluation processes discussed during the evaluation technical assistance (TA) sessions
How the climate change bundle follows evaluation best practices discussed during the evaluation TA sessions
How the evaluation TA sessions directly apply to the climate change bundle
Visualize how to develop a long-term research agenda for your program.
EVALUATION CAPACITY BUILDING OBSERVATION FORM
Observer name: ______________________________
Date of observation: ___________________________
Presentation title: _____________________________
Number of participants: ________________________
Overall, how satisfied are you that this presentation met the learning goals/objectives?
Very Dissatisfied, Dissatisfied, Unsure, Satisfied, Very Satisfied
How much would you agree or disagree with the following statements?
(Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree; notes for each item)
The instructor(s) clearly explained the material covered in this module.
The activities were helpful to participants’ understanding of the material.
The answers the instructor(s) gave to participants’ questions were clear.
The instructor(s) provided illustrative examples.
The exercises in the presentation were well-facilitated.
The participants were engaged in the module.
The participants learned or improved upon the objectives set in the module.
How does the course help develop content knowledge, skills, or thinking practices related to these learning objectives? _______________________________________________________________________
What are the strengths of this learning experience? _______________________________________________________________________
Do you have any suggestions for how to improve this learning experience? _______________________________________________________________________
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Rosemarie L. O'Conner |
File Modified | 0000-00-00 |
File Created | 2023-09-20 |