1 Volunteer Generation Fund Instruments

AmeriCorps Program Life Cycle Evaluation—Volunteer Generation Fund Grant Program Evaluation

20200716 life cycel evaluation VGF Instruments

OMB: 3045-0195

Document [docx]
Download: docx | pdf

Shape3

Shape2 Shape1

Submitted to:
AmeriCorps
250 E. St., SW

Washington, DC 20024

Submitted by:
ICF Incorporated, LLC

9300 Lee Highway

Fairfax, VA 22031
https://www.icf.com

May 3, 2021

Supporting Statement for Paperwork
Reduction Act Submissions

Volunteer Generation Fund (VGF)
Grant Program Bundled Evaluation

Instruments


CONTENTS



STATE SERVICE COMMISSION (GRANTEE)
INTERVIEW 1 PROTOCOL

Hello! Thank you for meeting with me today. My name is XX, and I’m with ICF, a research and evaluation organization working with AmeriCorps to understand how states utilize the Volunteer Generation Fund (VGF) grant. Today, we will be asking about ways in which this funding has been used by [state commission name] to engage in [VGF-funded activity, e.g., capacity building services] to support organizations in your state to recruit, manage, and support volunteers. Your response will help AmeriCorps to improve its VGF grant program.

This discussion is voluntary. You can decline to answer any questions. You can stop participating at any time. The information you share will remain private during and after the project. Neither your name nor any personal information about you will be shared beyond the ICF research team. We will maintain discussion data in secure areas. I would like to record the discussion to document information accurately. Only the ICF research team will have access to the recording. If you choose not to be recorded, I will take notes instead. I will not include your name in the notes.

(Confirm permission to record the session and, if confirmed, begin the recording.)

Before we begin, I want to check with you again that it is okay to record today’s discussion.

Commission Role and Model

  1. Let’s begin with some background information. Please tell me a little bit about your role with the [state name] Commission and your role as it relates to the Volunteer Generation Fund grant.

    1. Who are other key commission staff who help manage VGF grant activities, and what are their roles as it relates to the VGF grant?

  2. Can you please share the overarching approach that your state commission is implementing to develop or support community-based entities to recruit, manage, and support volunteers and increase the number of volunteers in your state?

(Note: Interviewer should have a good understanding of the commission’s VGF program components and model, and should probe if any of these components are not mentioned.)

    1. What approach does your state use to build capacity of volunteer connector organizations (or other entities) to increase volunteer recruitment, retention, and support of volunteers?

    2. What approach does your state use to create and improve volunteer networks within the state?

    3. What other components of your state’s model are aimed at improving volunteer recruitment (or increasing the number of volunteers in your state)?

    4. What approaches does your state use to engage targeted populations in volunteer opportunities (e.g., corporate employees, students/youth, seniors/retirees, skilled volunteers in disaster management or other professions)?

  1. Did the commission previously receive a VGF grant?

    1. If yes, when? What activities were implemented during previous grant cycles?

    2. If no, what components of the model were implemented prior to receiving the fiscal year (FY) 2020 VGF grant?

  2. What are the main goals and objectives that the commission is trying to achieve during the current VGF grant term as it relates to capacity building and increasing the number of volunteers in your state? (If not discussed previously.)

    1. What changes, if any, have been made since your FY2020 application?

      1. What prompted these changes?

    2. How, if at all, have you made changes based on the COVID-19 pandemic?

    3. How, if at all, have you made changes to your approach to address diversity and inclusion?

    4. What changes, if any, have been made to the timeline of VGF-funded activities since your FY2020 application?

      1. What prompted these changes?

    5. What VGF activities, if any, have been accomplished to date since receiving the FY2020 grant?

    6. What are the key VGF-funded activities, if any, that will take place over FY2022?

  3. What data do you collect to assess your performance measures outputs and outcomes as it relates to capacity building or number of volunteers in your state?

    1. How do you collect this data?

    2. What other data do you collect, if any?

  4. Are you working with a third party evaluator or an internal evaluator to evaluate your program model?

    1. If yes, please describe your evaluation design/activities to date, as well as future plans for evaluation activities.

Program Implementers

(Note: These questions focus on the role of program implementers, which may include commission staff, program partners, grantees, consultants, or other entities.)

  1. What is the commission’s role, if any, in implementing components of the model? (Interviewer should have some idea if the commission plays a role; if the commission is the implementer, skip questions about program partners.)

  2. Please tell me about your program partners that are responsible for implementing [VGF role/activity], if you have any.

  3. How many program partners do you work with in implementing [VGF role/activity]? (If not already stated.)

  4. How were these program partners selected?

    1. (If RFP): Can you tell me a little about the RFP process to select your partners/grantees?

    2. How long have you worked with your partners? (e.g., Were partnerships established prior to the VGF grant?)

  5. What model do program implementers utilize to provide the training?

    1. Please describe the role of the program implementers (or partners) in developing the program activities and timeline.

      1. Please describe the role of the participants (beneficiaries/volunteer organizations).

    2. If program implementers utilize a train-the-trainer model, please describe who provides the training to the trainer.

      1. Please describe the role of the trainer and who participates in their training sessions.

  6. Are program implementers developing the curriculum, training, or capacity building activities as part of the VGF grant, or are they implementing an already existing program?

    1. Please describe what aspects of the current program were developed prior to this grant. (If not previously discussed.)

  7. What VGF-funded activities do your program implementers do to increase volunteer networks in your state, if any? (If not already discussed.)

  8. How do program implementers work with other targeted populations (e.g., corporations, schools, community-based youth organizations, or skilled volunteers)? (If not already discussed. Note: This may not be relevant to all grantees.)

  9. How, if at all, have program implementers’ activities/trainings been impacted by COVID-19?

    1. What COVID-19-specific volunteer management training, if any, has been provided?

    2. What challenges, if any, have program implementers encountered due to the pandemic?

  10. How, if at all, have any program implementers’ activities/volunteer management trainings included a focus on diversity and inclusion as it relates to volunteer management?

  11. Are program implementers required to collect and assess VGF goals/objectives, outputs, and outcomes?

    1. If yes, what type of data do they collect?

    2. Are program implementers required to report on any performance measures? If so, which ones?

  12. What data or reports do program implementers submit to the commission? How often do program implementers submit data or reports?

Program Beneficiaries

  1. How, if at all, do commission staff interact with VGF-funded program recipients such as volunteer organizations, corporations, schools, or community-based organizations?

  2. What feedback, if any, have you received from program recipients about their experiences working with the program implementers?

  3. What types of data do program beneficiaries collect to help either program implementers or the state commission assess VGF goals/objectives, outputs, and outcomes, if any?

    1. How do they collect this data?

  4. [If the state commission implements other program components (e.g., platform/portal) related to volunteer recruitment or retention, in addition to capacity building/training, we can ask about the state commission’s perceived awareness of how program beneficiaries utilize these other components.]

  5. We’d like to speak to one or two program implementers. Do you have any recommendations?

  6. What additional feedback or insights, if any, would you like to share with us regarding the VGF program?

STATE SERVICE COMMISSION (GRANTEE)
INTERVIEW 2 PROTOCOL

Hello! Thank you for meeting with me today. My name is XX, and I’m with ICF, a research and evaluation organization working with AmeriCorps to understand the impacts of its Volunteer Generation Fund (VGF) grant program. We appreciate the time you have already taken to meet with us and tell us about [state commission name’s] VGF grant-funded activities. Today, we will be asking follow-up questions about ways in which this funding has been used by [state commission name] to engage in [VGF-funded activity, e.g., capacity building services] to support organizations in your state to recruit, manage, and support volunteers, and any initial outcomes. Your response will help AmeriCorps to improve its VGF grant program.

This discussion is voluntary. You can decline to answer any questions. You can stop participating at any time. The information you share will remain private during and after the project. Neither your name nor any personal information about you will be shared beyond the ICF research team. We will maintain discussion data in secure areas. I would like to record the discussion to document information accurately. Only the ICF research team will have access to the recording. If you choose not to be recorded, I will take notes instead. I will not include your name in the notes.

(Confirm permission to record the session and, if confirmed, begin the recording.)

Before we begin, I want to check with you again that it is okay to record today’s discussion.

  1. Let’s begin with some background information. We last spoke on [insert date]. At that time, you noted that your role was [role with commission and role with VGF from previous interview]. How, if at all, has your role changed?

    1. You noted that there are also key staff who are responsible for [commission and VGF roles]. How, if at all, have those staff roles changed?

  2. What changes, if any, have been made to the model since we last spoke?

    1. What prompted these changes?

  3. What are the key activities that took place since we last spoke?

  4. What changes, if any, have been made to the timeline of VGF grant activities?

    1. What prompted these changes?

  5. How would you describe the overall impact of the VGF grant to date?



  1. What are some of the successes you’ve had at the commission level related to the VGF grant and capacity building? What about other VGF-funded activities?

(Probe: How might successes shape remaining grant activities or activities beyond the grant?)

  1. What are some of the challenges you’ve had at the commission level related to the VGF grant and capacity building? What about challenges related to other VGF-funded activities?

    1. What are some lessons learned?

  2. (If the state proposed an evaluation in its proposal): Please provide an update of your evaluation activities conducted to date.

  3. What type of data do you (or your program implementers) collect related to outputs and outcomes associated with volunteer management practices?

    1. What other reports, if any, have you received from program implementers related to program activities and outcomes?

    2. What have been the key outcomes to date?

  4. What are some of the successes related to [insert program implementer names]’s VGF-funded grant activities, if any?

  5. What are some of the challenges related to [insert program implementer names]’s VGF-funded grant activities, if any?

    1. What are some lessons learned?

  6. What are some of the successes, if any, related to program beneficiaries implementing volunteer management strategies learned through the training/capacity building?

  7. What are some of the challenges related to program beneficiaries implementing volunteer management strategies learned through the training/capacity building, if any?

    1. What are some lessons learned?

  8. To what extent are program beneficiaries engaged in volunteer networks, if at all? (If the state has one, and if not already discussed.)

  9. To what extent are program beneficiaries utilizing the volunteer portal/platform, if at all? (If the state has one, and if not already discussed.)

  10. To what extent have you (the state commission) met your goals related to:

    1. Number of volunteers (or new volunteers)

    2. Volunteer organizations receiving capacity building services

    3. Volunteer organizations increasing effectiveness or efficiency in volunteer management

    4. Retention of volunteers

    5. Establishment or expansion of volunteer networks

    6. Implementation of programs with targeted populations (e.g., schools, corporations, youth programs)

  11. What challenges, if any, do you foresee in meeting your (the state commission’s) capacity building and volunteer goals by the end of the grant term? (If not already discussed.)

  12. Over the past year, how, if at all, has the pandemic impacted any of your programming?

    1. What challenges, if any, have program implementers reported?

  13. Over the past year, how, if at all, have you incorporated diversity and inclusion into your programming?

  14. (Based on the state commission’s model, we may ask additional questions related to outcomes.)

  15. In what ways, if any, has your VGF grant improved volunteer organizations’ capacity to utilize volunteers?

  16. What else can the state commission do to better support volunteer organizations with regard to volunteer recruitment, management, and retention?

  17. What additional feedback or insights, if any, would you like to share with us regarding the VGF program?



PROGRAM IMPLEMENTER INTERVIEW PROTOCOL

Hello! Thank you for meeting with me today. My name is XX, and I’m with ICF, a research and evaluation organization working with AmeriCorps to understand the impacts of its Volunteer Generation Fund (VGF) grant program. Today, we will be asking about ways in which this funding has been used by [state commission name and/or program implementer organization name] to engage in [VGF-funded activity, e.g., capacity building services] to support community-based entities to recruit, manage, and support volunteers. Your response will help AmeriCorps to improve its VGF grant program.

This discussion is voluntary. You can decline to answer any questions. You can stop participating at any time. The information you share will remain private during and after the project. Neither your name nor any personal information about you will be shared beyond the ICF research team. We will maintain discussion data in secure areas. I would like to record the discussion to document information accurately. Only the ICF research team will have access to the recording. If you choose not to be recorded, I will take notes instead. I will not include your name in the notes.

(Confirm permission to record the session and, if confirmed, begin the recording.)

Before we begin, I want to check with you again that it is okay to record today’s discussion.

  1. Let’s begin with some background information. Please tell me a little bit about your role at [program implementer] and your role as it relates to the Volunteer Generation Fund grant activity, [main activity of program implementer].

    1. Who are the other staff, if any, that are involved in the VGF-funded capacity building activities and what are their roles?

  2. How do you select volunteer organizations to participate in the volunteer management training/capacity building?

    1. How competitive, if at all, is the process to be able to participate?

    2. What type of model do you use for training one or more organizations? (e.g., cohort model)

    3. How many volunteer organizations have you trained as part of this VGF grant to date since October 2020?

    4. How many more volunteer organizations do you plan to train by September 2023 as part of this VGF grant?

  3. Please describe the organizations that participate in the training and any requirements these organizations must meet. (For example, types of organizations may include volunteer connector organizations, government entities, volunteer organizations, non-profits, community-based organizations, or corporations.)

  4. Please describe the capacity building or training that you provide these organizations.

    1. What curriculum or program do you use?

    2. Who or what organization developed the curriculum/training?

      1. Did your organization develop the curriculum/training?

      2. If so, did your organization work with any others to develop the curriculum/training? Was the curriculum/training a part of the current VGF grant?

    3. How many capacity building practices do you typically cover in the training?

      1. How many practices or strategies do you generally cover in each training session?

    4. What materials do you provide to training participants?

    5. How many sessions/trainings does each organization participate in?

      1. What are requirements related to training?

        1. How many sessions or hours are participants required to attend, if there are any requirements?

        2. How many staff from each organization are required to attend, if there are any requirements?

    6. What other expectations do you have for the training participants (e.g., do they have to outline a plan for implementing capacity building practices)?

    7. What certifications, if any, do participants or organizations receive?

      1. What are the requirements for certification?

  5. What partners, if any, do you work with to provide training or capacity building?

    1. What is the role of your partners?

    2. Does your program use the train-the-trainer model?

      1. If yes, please describe.

  6. Since October 2020, what are the greatest needs of the organizations with regard to capacity building?

    1. To what extent do volunteer organizations need help with recruitment strategies (e.g., finding more volunteers, finding skill-based volunteers, etc.)?

    2. To what extent do volunteer organizations need help with volunteer engagement or retention strategies?

  7. How, if at all, have you incorporated COVID-19-specific volunteer management training or capacity building into the training you provide organizations?

  8. How, if at all, have you incorporated diversity and inclusion into your volunteer management training or capacity building?

  9. Since October 2020, what data, if any, do you collect from the volunteer organizations prior to their participation in the program? (Examples of data collected may include information about organizations’ needs or what they are hoping to learn from the training.)

    1. How do you collect this data?

  10. Since October 2020, what data, if any, do you collect from the volunteer organizations after they participate in the training?

    1. How do you collect feedback regarding participant experiences with the program?

      1. What have they liked about the training?

      2. What suggestions do they have for improvement?

    2. Do you collect data on the extent to which they implement volunteer management practices?

      1. What strategies do organizations report successfully implementing?

      2. What strategies do organizations find challenging?

    3. What data do you collect from volunteer organizations as it relates to their recruitment and/or retention? (Examples of data collected may include tracking the number of volunteers before participating in training and then tracking volunteer numbers over time.)

  11. How did your organization become involved in this grant? Has your organization worked with the state commission before? If so, for how long and in what area?

  12. Since October 2020, how often do you communicate with the state commission regarding your progress?

    1. To what extent do you share your successes, challenges, and lessons learned with the state commission, if at all?

  13. Since October 2020, what data or reports do you provide to the state commission as it relates to capacity building outputs and outcomes?

    1. How often do you provide the state commission with data?

  14. What other data from training participants/organizations do you collect, if any?

  15. How do you work with a third party or an internal evaluator to evaluate your program, if at all?

    1. Please describe your evaluation design and activities.

  16. What do you see as the biggest success of your program to date since October 2020?

  17. What are the biggest challenges of your program to date since October 2020?

    1. What are some lessons learned?

  18. What are some of the successes related to program beneficiaries implementing volunteer management strategies learned through the training/capacity building?

  19. What are some of the challenges related to program beneficiaries implementing volunteer management strategies learned through the training/capacity building?

    1. What are some lessons learned?

  20. What challenges, if any, have program beneficiaries encountered as a result of the COVID-19 pandemic?

  21. (If a program is responsible for creating/expanding a network): What is your role in building a volunteer network with the state? How does your organization participate in the state’s volunteer network?

  22. (If a state has a volunteer portal or platform): How does your organization utilize the volunteer platform or portal?

  23. In what ways, if any, has the [program implementer’s activities] improved volunteer organizations’ capacity to utilize volunteers?

  24. What additional feedback or insights, if any, would you like to share with us regarding the VGF program?

BENEFICIARY FOCUS GROUP PROTOCOL

Hi, everyone. Thank you for being with us today. My name is XX, and I’m with ICF, a research and evaluation organization working with AmeriCorps to understand the impacts of its Volunteer Generation Fund (VGF) grant. Today, we will be asking about ways in which this funding has been used by the [state name] State Commission and/or [program implementer] to support your organization by increasing volunteer recruitment and/or improving volunteer management practices. Your response will help AmeriCorps to improve its VGF grant program.

This discussion is voluntary. You can decline to answer any questions. You can stop participating at any time. The information you share will remain private during and after the project. Neither your name nor any personal information about you will be shared beyond the ICF research team. We will maintain discussion data in secure areas. I would like to record the focus group to document information accurately. Only the ICF research team will have access to the recording.

(Confirm permission to record the session and, if confirmed, begin the recording.)

Before we begin, I want to check with you again that it is okay to record today’s focus group discussion.

Let’s begin with some brief introductions.

  1. Please tell me your role in your organization, your role in [VGF training activity], and how long you’ve been with your organization.

  2. Please describe how your organization has participated in [VGF training activity].

    1. How many staff from your organization participated in training? (e.g., Just the focus group participant or several staff?)

  3. What are the different components of [VGF training activity]? (Probe: webinars, workshops, conferences, online modules, lectures, etc.)

    1. How many hours was each training/session?

    2. How often did your organization attend trainings/sessions? (Probe: intensity of the training)

    3. What resources or training materials have you received, if any? (Probe: slide decks, training manuals, etc.)

  4. What training program or curriculum was used, if any? (e.g., Service Enterprise certification, Volunteer Impact Leadership Training certification)

    1. Did you, one of your colleagues, or your organization receive certification as a result of training? What did this certification entail?

Volunteer Recruitment and Selection

  1. What training or capacity building did you receive on volunteer recruitment and selection?

    1. What aspects of volunteer recruitment and selection training, if any, were helpful?

    2. What aspects of volunteer recruitment and selection training, if any, could be improved?

  2. What volunteer recruitment strategies have you learned, if any, that focused on recruiting volunteers during the beginning of the pandemic?

  3. What challenges, if any, have you faced with regard to recruiting volunteers during the pandemic?

  4. What volunteer management strategies have you learned, if any, that focus on recruiting diverse volunteers (or what strategies related to diversity and inclusion have you learned)?

  5. What challenges, if any, have you faced with regard to recruiting diverse volunteers (or what challenges do you face with developing strategies that focus on recruiting diverse volunteers)?

  6. What were the most important volunteer recruitment strategies you learned, if any, through your participation in training?

  1. What strategies, if any, worked particularly well for your organization?

  1. What were the least important volunteer recruitment strategies you learned, if any, through your participation in training?

  1. What strategies, if any, didn’t work for your organization?

  1. How have your organization’s volunteer recruitment and selection processes changed since the training, if at all?

    1. How has training improved your organization’s ability to recruit volunteer applicants and select volunteers, if at all?

    2. What changes, if any, have you noticed in the number of volunteers recruited?

    3. What changes, if any, have you noticed in the characteristics of volunteers recruited and selected? (Probe: demographics, quality, and/or matching of skill sets)



Volunteer Training and Management

  1. What training or capacity building did you receive on volunteer management and retention?

    1. What aspects of volunteer management training, if any, were helpful?

    2. What aspects of volunteer management training, if any, could be improved?

  2. What were the most important volunteer management strategies you learned, if any, through your participation in training?

  1. What strategies, if any, worked particularly well for your organization?

  1. What were the least important volunteer management strategies you learned, if any, through your participation in training?

  1. What strategies, if any, didn’t work for your organization?

  1. How have your organization’s volunteer management processes changed, if at all, since the training?

    1. How has training improved your organization’s ability to manage volunteers, if at all?

    2. What changes, if any, have you noticed in volunteer engagement and satisfaction?

    3. What changes, if any, have you noticed in volunteer retention?

      1. What about the changes in the characteristics of volunteers retained?

  2. What volunteer management strategies have you learned, if any, that focused on managing volunteers during the beginning of the pandemic?

  3. What challenges, if any, have you faced with regard to managing, supporting, or retaining volunteers during the pandemic?

  4. What volunteer management strategies have you learned, if any, that focus on managing diverse volunteers?

  5. What challenges, if any, have you faced with regard to managing, supporting, or retaining diverse volunteers?

Data, Platforms, and Networks

  1. What type of volunteer data, if any, does your organization collect? (Probe: hours worked, satisfaction, retention)

    1. How does your organization use data to make decisions with regard to volunteer management?

    2. What type of data, if any, does your organization collect on volunteer recruitment?

(If the state has an online volunteer platform or portal funded by VGF):

  1. What is your awareness of [name of state volunteer platform]?

  2. To what extent does your organization utilize this platform for recruitment?

  3. How helpful, if at all, has this platform been for recruiting volunteers?

(If the state uses VGF funds to create or enhance a volunteer network):

  1. What is your awareness of [name of state volunteer organization network]?

  2. To what extent does your organization participate in this network?

  3. How has this network supported your organization’s volunteer management efforts?

    1. What support was effective? In what ways was it effective?

    2. What could be improved? In what ways could it be improved?

Concluding Topics

  1. In what ways, if any, has the support from [program implementer’s activities, and if applicable, volunteer portal/platform and/or the state’s VGF-funded volunteer network] improved your organization’s capacity to utilize volunteers?

  2. What additional support do you need with regard to volunteer recruitment, management, or support?

  3. What additional feedback or insights would you like to share with us regarding your participation in [program implementer’s activities, and if applicable, volunteer portal/platform, and/or the state’s VGF-funded volunteer network]?



BENEFICIARY SURVEY

Thank you for taking the time to complete this survey. ICF, a research and evaluation organization, is partnering with AmeriCorps to understand the impact of its Volunteer Generation Fund (VGF) grant. In this survey, we will be asking about ways in which this funding has been used by the [state name] State Commission and/or [program implementer] to support your organization through volunteer management training and other volunteer management support. Your response will help AmeriCorps to improve its VGF grant program.

  1. In which state do you work? (drop-down)

  2. For which organization in [state name] do you work? (drop down)

  3. What is your current role at [organization name]?

    1. Executive director/CEO/president

    2. Volunteer manager/coordinator

    3. Other (please specify): __________

Volunteer Management Training

  1. What type of volunteer management training/capacity building activities are available to your organization through the [state name] State Commission or [program implementer]? (matrix; Yes/No/Don’t know)

    1. Interactive training sessions or workshops (in-person or virtual)

    2. Conferences (in-person or virtual)

    3. One-on-one training (training included only your organization staff with program implementer trainers)

    4. Webinars

    5. Coaching

    6. Mentoring

    7. Online training modules (asynchronous)

    8. Other (please specify): __________

  2. When did your organization first participate in training through [the state commission or program implementer]?

    1. Prior to October 2020

    2. October 2020 – March 2021

    3. April 2021 – September 2021

    4. October 2021 – March 2022

    5. April 2022 – Present

  3. Since October 2020, how many other individuals at your organization have participated in training/capacity building activities through the [state name] State Commission or [program implementer]? (drop-down 0-10+)

  4. Since October 2020, approximately how many hours of volunteer management training have you received? (drop-down 0-20+)

  5. Do you or members of your organization still plan to complete additional training through [the state commission or program implementer]?

    1. Yes

    2. No

    3. Undecided/don’t know

  6. How many more hours of training do you or members of your organization plan to complete with [the state commission or program implementer] in total? (drop-down 0-10+)

  7. What training curriculum was used for your training? (Select all that apply.)

  1. Service Enterprise Initiative

  2. Pathways of Public Service and Civic Engagement

  3. FEMA courses

  4. California Specialized Training Institute courses

  5. Points of Light Volunteer Manager training curriculum

  6. State commission-developed curriculum



  1. Other (please specify): __________

  2. Don’t know

  1. What certification has your organization received since October 2020, if any? (Select all that apply.)

    1. Service Enterprise certification

    2. Service Enterprise Hub (qualification to train other organizations)

    3. State commission-developed certification

    4. Other (please specify): __________

    5. My organization has not received any certification as of [insert month/date of survey]

  2. What certification might your organization receive by September 2023, if any? (Select all that apply.)

    1. Service Enterprise certification

    2. Service Enterprise Hub (qualification to train other organizations)

    3. State commission-developed certification

    4. Other (please specify): __________

    5. My organization will not receive any certification

  3. What certification did you or one of your staff members receive, if any, since October 2020? (Select all that apply.)

  1. Certification in Volunteer Administration (United Way)

  2. State commission-developed certification

  3. Other (please specify): __________

  4. Neither I nor any staff members received a certification

  1. What certification do you expect that you or one of your staff members will receive by September 2021, if any? (Select all that apply.)

  1. Certification in Volunteer Administration (United Way)

  2. State commission-developed certification

  3. Other (please specify): __________

  4. Neither I nor any staff members will receive a certification

  1. Since October 2020, for which of the following practices did you receive training through [program implementer]? (Select all that apply.)

    1. Market research and community needs assessments

    2. Strategic planning to maximize volunteer impact

    3. Recruiting and marketing to prospective volunteers

    4. Interviewing, screening, and selecting volunteers

    5. Orienting and training volunteers

    6. Ongoing supervision and management

    7. Recognition and volunteer development

    8. Measuring outcomes and evaluating the process

  2. How helpful was the training for learning volunteer recruitment and selection strategies, if at all? (Likert-type 5-point scale; unipolar; “Not at All Helpful,” “Slightly Helpful,” “Moderately Helpful,” “Very Helpful,” “Extremely Helpful”)

  3. To what extent did you implement the strategies you learned for volunteer recruitment and selection, if at all? (Likert-type 5-point scale; unipolar; “No Strategies,” “A Few Strategies,” “Several Strategies,” “Most Strategies,” “All Strategies” “Did Not Implement Any Strategies”)

  4. How has implementing the strategies for recruiting and selecting volunteers improved volunteer recruitment, if at all? (Likert-type 5-point scale; unipolar; “Not at All Helpful,” “Slightly Helpful,” “Moderately Helpful,” “Very Helpful,” “Extremely Helpful”)

  5. How helpful was the training for learning volunteer management and support strategies, if at all? (Likert-type 5-point scale; unipolar; “Not at All Helpful,” “Slightly Helpful,” “Moderately Helpful,” “Very Helpful,” “Extremely Helpful”)

  6. To what extent did you implement the strategies you learned for volunteer management and support, if at all? (Likert-type 5-point scale; unipolar; “No Strategies,” “A Few Strategies,” “Several Strategies,” “Most Strategies,” “All Strategies”)



  1. How has implementing the strategies for managing and supporting volunteers improved volunteer engagement or retention, if at all? (Likert-type 5-point scale; unipolar; “Not at All Helpful,” “Slightly Helpful,” “Moderately Helpful,” “Very Helpful,” “Extremely Helpful,” “Did Not Implement Any Strategies” )

  2. To what extent do you or your organization need additional training or support on the following practices, if at all? (Matrix; Likert-type 5-point scale; unipolar; “None at All,” “A Little,” “A Moderate Amount,” “A Lot,” “A Great Deal”)

    1. Market research and community needs assessments

    2. Strategic planning to maximize volunteer impact

    3. Recruiting and marketing to prospective volunteers

    4. Interviewing, screening, and selecting volunteers

    5. Orienting and training volunteers

    6. Ongoing supervision and management

    7. Recognition and volunteer development

    8. Measuring outcomes and evaluating the process

  3. How many of the individuals who volunteer with your organization work every month, on average?

    1. Less than 10 persons

    2. 10-19 persons

    3. 20-29 persons

    4. 30-49 persons

    5. 50-100 persons

    6. 100+ persons

  1. Have you or anyone at your organization used the [state commission portal/platform name] to help search for or identify potential volunteers?

    1. Yes

    2. No

    3. Don’t know

  2. How many potential volunteers did you identify through your searches? (drop-down 0-20+)

  3. Have you or anyone at your organization used the [state commission portal/platform name] to post volunteer opportunities with your organization?

    1. Yes

    2. No

    3. Don’t know

  4. How many volunteer opportunities did you (or someone at your organization) post? (drop-down 0-20+)

  5. Has your number of volunteers increased as a result of volunteer recruitment training, resources, or assistance from the [state name] State Commission or [program implementer]?

  1. Yes

  2. No

  3. Don’t know

  1. To what extent, if at all, have the characteristics of volunteers changed in a way that aligns with your expectations (e.g., targeted demographic of volunteers or volunteer skill sets) as a result of training, resources, or assistance from the [state name] State Commission or [program implementer]? (Likert-type 5-point scale; unipolar; “Not at All,” “Slightly,” “Moderately,” “Very,” “Extremely”)

  2. To what extent, if at all, has support from the [state name] State Commission or [program implementer] improved your organization’s process of matching volunteers to volunteer opportunities? (Likert-type 5-point scale; unipolar; “Not at All,” “Slightly,” “Moderately,” “Very,” “Extremely”)

  3. To what extent, if at all, has support from the [state commission portal/platform name, if applicable] improved your organization’s process of matching volunteers to volunteer opportunities? (Likert-type 5-point scale; unipolar; “Not at All,” “Slightly,” “Moderately,” “Very,” “Extremely”)

  4. What additional feedback or insights would you like to share with us regarding your participation in [program implementer]’s program?



VOLUNTEER FOCUS GROUP PROTOCOL

Hi, everyone. Thank you for being with us today. My name is XX, and I’m with ICF, a research and evaluation organization working with AmeriCorps to understand the impacts of its Volunteer Generation Fund (VGF) grant. Today, we will be asking about your experiences as a volunteer with [organization]—in particular how [organization] recruited you as a volunteer and how its staff currently engages with you as a volunteer. Your response will help AmeriCorps to improve its VGF grant program and how it supports the training provided to organizations that utilize volunteers.

This discussion is voluntary. You can decline to answer any questions. You can stop participating at any time. The information you share will remain private during and after the project. Neither your name nor any personal information about you will be shared beyond the ICF research team. We will maintain discussion data in secure areas. I would like to record the focus group to document information accurately. Only the ICF research team will have access to the recording.

(Confirm permission to record the session and, if confirmed, begin the recording.)

Before we begin, I want to check with you again that it is okay to record today’s focus group discussion.

Let’s begin with some brief introductions.

  1. Please tell me how long you have been a volunteer with [organization] and what your role is as a volunteer.

Volunteer Recruitment and Selection

  1. How did you hear about the volunteer opportunity?

    1. Did you look for opportunities in your state’s portal or platform?

    2. Did you see an announcement on your local TV station or hear on the radio about volunteering in general?

    3. Did you hear about opportunities through an event? Was it virtual or in-person?

  2. What were you looking for with regard to a volunteer opportunity?

  3. What were the various reasons that contributed to your decision to volunteer with [organization]?



Volunteer Management

  1. When you first became a volunteer, what training did [organization] provide?

    1. What were other onboarding activities?

  2. What were your initial volunteer activities?

    1. How often did you volunteer (e.g., hours/day per week)?

    2. How have your volunteer activities changed, if at all?

    3. To what extent did your volunteer activities align with what you were expecting?

      1. Were you matched for a specific volunteer position based on your skill set?

      2. To what extent was your role clearly defined? How clearly defined is your role now?

  3. When you first became a volunteer, who oversaw your volunteer activities?

    1. How often did you meet with your volunteer manager?

    2. How has this changed, if at all?

  4. To what extent were you able to provide feedback to your volunteer manager about your experience as a volunteer?

    1. How has this changed, if at all?

  5. What mentorship opportunities, if any, did [organization] provide when you first started?

    1. How as this changed, if at all?

  6. To what extent have you developed relationships with others at [organization]?

    1. How has [organization] promoted relationship building among volunteers or between volunteers and employees?

  7. When you started with [organization], to what extent did [organization] provide professional and/or emotional support?

    1. How has this changed, if at all?

  8. What learning activities, if any, have been provided through [organization]?

    1. (Probe for examples.)

  9. To what extent has [organization] recognized contributions of volunteers?

    1. How has [organization]recognized your contributions?

  10. How has COVID-19 impacted your volunteer duties?

    1. (Probe for examples.)

  11. How has COVID-19 impacted how you engage with [organization]?

    1. (Probe for examples.)

  12. What type of support has [organization] provided since the beginning of COVID-19?

    1. (Probe for examples.)

  13. To what extent does [organization] promote diversity and inclusion among volunteers and employees?

    1. (Probe for examples.)

  14. How would you describe your overall satisfaction with volunteering with [organization]?

  15. How long do you plan to continue to volunteer with [organization]?

    1. (If planning to stay): What factors contribute to your decision to want to stay with [organization]?

    2. (If planning to leave in the near future): What factors contribute to your decision to want to leave [organization]? What, if anything, could [organization] do to make you change your mind to stay?

Concluding Topics

  1. What additional support do you need from [organization]?

  2. In what ways could [organization] improve how it recruits volunteers?

  3. In what ways could [organization] improve how it trains and/or supports volunteers?

  4. What additional feedback or insights would you like to share with us regarding your experience volunteering with [organization]?



EVALUATION TECHNICAL ASSISTANCE
POST-PRESENTATION SURVEYS

Module 1 — Presentation 1: Introduction and Evaluation Basics

Knowledge Questions

  1. Which of the following is NOT a good reason to conduct a program evaluation? (Select one.)

☐ a. Program A wants to demonstrate the benefits of the program to stakeholders.

☐ b. Program B wants to generate a daily report for its employees.

☐ c. Program C wants to understand what needs to be improved in the program.

☐ d. Program D wants to make decisions about resource allocation.

2. Please match the following terms with the right definition. (Click “Choose an item” to select the letter that corresponds to the correct definition for each term.)

ANSWER

TERMS

DEFINITIONS

Evidence

a. The reasons why you think your program will achieve its goals

Evaluation

b. Systematic investigation about your
program

Quantitative data

c. Numerical information that can be counted, quantified, and mathematically analyzed

Qualitative data

d. Narrative information, often describing context, motivations, and how and why answers

  1. Which of these statements best highlights the difference between program evaluation and performance measurement? (Select one.)

☐ a. Performance measurement uses a systematic approach; evaluation uses regular updates of changes over time.

☐ b. Performance measurement answers program-level research questions; evaluation answers routine questions.

☐ c. Performance measurement reports data that require programmatic change;

evaluation reports data that require immediate change.

☐ d. Performance measurement indicates whether a project is moving

according to plan; evaluation systematically examines the workings of a

program.



Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ Very Dissatisfied

☐ Dissatisfied

☐ Unsure

☐ Satisfied

☐ Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    5. I understand how to apply this material to my own program.

  2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Why evaluations can be useful.

    2. The difference between evidence and data.

    3. The difference between evaluation and performance measurement.

  1. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.


Module 1 — Presentation 2: Theory of Change and Logic Models

Knowledge Questions

  1. Which of these best defines what a theory of change is? (Select one.)

☐ a. An explanation of any negative or positive impact your program has on the community

☐ b. An explanation of what your program’s ultimate goal is

☐ c. An example of how you’d like to change your role within your program

☐ d. A reflection on how you achieve your personal goals every day within your program

  1. Which of the following is NOT a reason to develop a theory of change? (Select one.)

☐ a. It ensures people will not ask you to evaluate your program.

☐ b. It forces your organization to articulate the assumptions underlying your

program.

☐ c. It highlights the goals your program is trying to achieve.

☐ d. It shows the unique contribution your program makes to achieve its goals.

  1. Select the statement that best describes how a logic model connects to your theory of change. (Select one.)

☐ a. A theory of change visually describes a logic model.

☐ b. A logic model visually shows how your program’s theory of
change works.

☐ c. A theory of change visually represents the connections between program components; a logic model is a statement describing the program’s goals.

☐ d. A logic model presents in one chart every theory of change that you can come up with.

  1. Using what you’ve learned in class about what goes into a logic model and the following theory of change, select one answer in questions a–f based on the following scenario.


There is not enough volunteerism in our state. We believe this is because of a need to expand volunteer capacity in our state. Our theory is that if we can leverage volunteers to achieve operational efficiency in areas they are most needed, then we will create a stronger volunteer infrastructure in our state. The strategic partnerships we have made with Service Enterprise organizations help with effectively using volunteers’ skills to address community needs; strengthen the volunteer network by promoting connections between non-profits, corporations, and volunteers; and train volunteers for additional skills they will need. By strengthening capacity building efforts and creating a volunteer network in the state, we will effectively use volunteers to enable organizations to accomplish their missions. Ultimately, this will increase our state’s volunteerism rates.

    1. An example of an input for this program would be:

☐ i. Funding for the program

☐ ii. Trainings provided to volunteers

☐ iii. Number of registered volunteers in the state

☐ iv. Expansion of volunteers’ skill sets

☐ v. Volunteerism rates

    1. An example of an activity for this program would be:

☐ i. Funding for the program

☐ ii. Trainings provided to volunteers

☐ iii. Number of registered volunteers in the state

☐ iv. Expansion of volunteers’ skill sets

☐ v. Volunteerism rates

    1. An example of an output for this program would be:

☐ i. Funding for the program

☐ ii. Trainings provided to volunteers

☐ iii. Number of registered volunteers in the state

☐ iv. Expansion of volunteers’ skill sets

☐ v. Volunteerism rates

    1. An example of a short-term outcome for this program would be:

☐ i. Funding for the program

☐ ii. Trainings provided to volunteers

☐ iii. Number of registered volunteers in the state

☐ iv. Expansion of volunteers’ skill sets

☐ v. Volunteerism rates

    1. An example of an intermediate outcome for this program would be:

☐ i. Funding for the program

☐ ii. Increased number of organizations that report increased effectiveness

☐ iii. Number of registered volunteers in the state

☐ iv. Expansion of volunteers’ skill sets

☐ v. Volunteerism rates

    1. An example of a long-term outcome for this program would be:

☐ i. Funding for the program

☐ ii. Increased number of organizations that report increased effectiveness

☐ iii. Number of registered volunteers in the state

☐ iv. Expansion of volunteers’ skill sets

☐ v. Volunteerism rates

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. What a theory of change is.

    2. How to come up with a theory of change for my program.

    3. What a logic model is.

    4. What the difference is between a logic model and a theory of change.

    1. How to figure out what goes into a logic model.

  1. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.




Module 1 — Presentation 3: Evaluation Planning

Knowledge Questions

  1. Which of the following is an example of a good evaluation question? (Select one.)

☐ a. Does the volunteer generation program work?

☐ b. Are Service Enterprise Hubs useful?

☐ c. To what extent do volunteerism rates differ for organizations

participating in this program compared to other organizations that did not participate?

☐ d. Does our program improve its volunteers’ skills?

  1. What’s the difference between process and outcome evaluations? (Select one.)

☐ a. Process evaluations examine how a program works; outcome

evaluations measure the results or effects of a program.

☐ b. Process evaluations measure changes in attitudes; outcome evaluations

examine whether the program has sufficient resources.

☐ c. Process evaluations measure changes to beneficiaries; outcome

evaluations measure which pieces of the program are working as expected.

☐ d. Process evaluations measure the results of a program; outcome evaluations

measure program activities.

  1. Which of the following is an example of a quantitative indicator for a process

evaluation? (Select one.)

☐ a. Students’ perceptions about what did or did not work well in their program

☐ b. AmeriCorps members’ opinions about what they liked and did not like

about their training

☐ c. The number of students who received tutoring

☐ d. Attributes of participants who completed all program components and

those who dropped out

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

  1. The instructor(s) clearly explained the material covered in this module.

  1. I felt comfortable voicing any questions I had in class.

  2. The activities were helpful to my understanding of the material.

  3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The difference between a process and an outcome evaluation.

    2. The attributes of a good evaluation question.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.




Module 2 — Presentation 1: Preparing to Collect Data

Knowledge Questions

  1. Which of the following is an example of primary data that can be collected to evaluate the impact of the recovery coach program? (Select one.)

☐ a. High school graduation rates from the U.S. Census Bureau

☐ b. Hospital records from the National Inpatient Sample

☐ c. Survey responses from volunteers in the program

☐ d. Employment information from the state

  1. When collecting data from the population, which of the following is a good reason to draw a sample? (Select all that apply.)

☐ a. It can be time consuming to collect data from an entire population.

☐ b. It costs less to collect data from a sample than a population.

☐ c. One can get a higher response rate with data collected from a sample

than from a population with follow-up efforts.

☐ d. A sample can be representative of a population.

  1. When should you collect the following types of data? (Fill in each box using the following options: before the intervention, during the intervention, after the intervention.)

a. Baseline data: Click or tap here to enter text. (before the intervention)

  1. Number of times adults have interacted with recovery coaches:

Click or tap here to enter text. (during or after the intervention)

  1. Changes in employment: Click or tap here to enter text. (after the intervention)

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The difference between primary data and secondary data.

    2. When data should be collected to evaluate a program.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 — Presentation 2: Data Collection Techniques

Knowledge Questions

  1. If you are intending to collect qualitative data from a participant, which data collection technique would you NOT utilize? (Select one.)

☐ a. Structured interviews

☐ b. Focus groups

☐ c. Structured observations

☐ d. Closed-ended survey questions

2. Read the questions below and identify the common pitfall (if any) associated with each of the survey questions. (Select one answer for each question.)

    1. Do you feel that you have had a great experience with the recovery coaches in the program?

☐ i. Double-barreled question

☐ ii. Leading question

☐ iii. Social desirability question

☐ iv. Non-applicable question

☐ v. No pitfall

    1. Please rate the quality of the monthly meetings with your coach, with 1 being “very useful” to 5 being “not useful at all.”

☐ i. Double-barreled question

☐ ii. Leading question

☐ iii. Social desirability question

☐ iv. Non-applicable question

☐ v. No pitfall











  1. Do you have any recommendations or complaints about your recovery coach?

☐ i. Double-barreled question

☐ ii. Leading question

☐ iii. Social desirability question

☐ iv. Non-applicable question

☐ v. No pitfall

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. When to use surveys for primary data collection.

    2. When to use focus groups for primary data collection.

    3. When to use interviews for primary data collection.

    4. When to use structured observations for primary data collection.

    5. The different types of data collection techniques.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.

Module 2 — Presentation 3: Data Analysis

Knowledge Questions

  1. For which question would quantitative data be used to answer? (Select one.)

☐ a. How many volunteers use the volunteer database daily?

☐ b. What do volunteers say about their experience with volunteer trainings?

☐ c. Why are some organizations better at using volunteers for COVID-19 recovery work versus others?

☐ d. What makes it harder for states to provide volunteers to rural areas?

  1. Which of the following is an example of qualitative analysis of data? (Select one.)

☐ a. Frequencies

☐ b. Counts

☐ c. Minimum and maximum

☐ d. Interpretation of direct quotes

  1. Which of the following is an effective way to quantitively analyze data? (Select one.)

☐ a. Measure whether a difference between two groups is statistically significant

☐ b. Code participant responses to analyze themes in the data

☐ c. Interpret differences in categories of coded responses

☐ d. Take narrative notes

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied



  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    5. I understand how to apply this material to my own program.

  2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Quantitative analysis techniques.

    2. Qualitative analysis techniques.

  3. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.



Module 3 — Presentation 1: Evaluation Reporting

Knowledge Questions

  1. Which of these questions should the discussion section of an evaluation report answer? (Select one.)

☐ a. What was the approach used to analyze the data?

☐ b. What was the theory of change for the program?

☐ c. Did the findings of the report support your theory of change?

  1. Which of these questions should the methodology section of an evaluation report answer? (Select one.)

☐ a. What was the approach used to analyze the data?

☐ b. What was the theory of change for the program?

☐ c. Did the findings of the report support your theory of change?

  1. Which of these questions should the introduction section of an evaluation report answer? (Select one.)

☐ a. What was the approach used to analyze the data?

☐ b. What was the theory of change for the program?

☐ c. Did the findings of the report support your theory of change?

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    1. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topic? (Click “Choose an item” to select the response that best corresponds to your answer.)

    1. What goes into an evaluation report.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.


Module 3 — Presentation 2: Using Evaluation for Program Improvement

Knowledge Questions

  1. Select all the ways you can use findings from a process evaluation for program improvement. (Select all that apply.)

☐ a. To demonstrate improvement for organizations after training

☐ b. To plan further evaluations to examine long-term increases in rates of

volunteerism

☐ c. To demonstrate changes in organizations’ attitudes after participation in

statewide volunteerism programs

☐ d. To change the duration of training the volunteers receive

☐ e. To revise the intervention model

☐ f. To change the program activities

  1. Select all the ways you can use findings from an outcome evaluation for program improvement. (Select all that apply.)

☐ a. To demonstrate improvements for organizations after training

☐ b. To plan further evaluations to examine long-term increases in rates of

volunteerism

☐ c. To demonstrate changes in organizations’ attitudes after participation in

statewide volunteerism programs

☐ d. To change the duration of training the volunteers receive

☐ e. To revise the intervention model

☐ f. To change the program activities

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied



  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.

    3. I felt comfortable voicing any questions I had in class.

    4. The activities were helpful to my understanding of the material.

    5. I understand how to apply this material to my own program.

  2. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Findings from a process evaluation.

    2. Findings from an outcome evaluation.

  3. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.




Module 3 — Presentation 3: Developing a Long-Term Research Agenda

Knowledge Questions

  1. Which of the following does NOT characterize a long-term research agenda? (Select one.)

☐ a. It should be progressive.

☐ b. Once it is established, it cannot be changed.

☐ c. It should build on what has been done in the past.

☐ d. It should point toward future goals.

  1. What needs to be done to build evidence to support the program goals over time? (Click “Choose an item” to arrange the following tasks in order of complexity, with “1” being the least complex.)

a. Build consensus about your overall evaluation goals.

b. Conduct a process evaluation.

c. Conduct an outcome evaluation.

d. Conduct an impact evaluation with a comparison group.

e. Expand program to multiple sites.

Satisfaction Questions

  1. Overall, how satisfied are you with this module? (Select one.)

☐ a. Very Dissatisfied

☐ b. Dissatisfied

☐ c. Unsure

☐ d. Satisfied

☐ e. Very Satisfied

  1. How much would you agree or disagree with the following statements? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The information I learned in this module was useful for me.

    2. The instructor(s) clearly explained the material covered in this module.


    1. I felt comfortable voicing any questions I had in class.

    2. The activities were helpful to my understanding of the material.

    3. I understand how to apply this material to my own program.

  1. After taking this module, how do you rate your understanding of the following topics? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. The key components of a long-term research agenda.

    2. How to develop a long-term research agenda for your program.

  2. What did you enjoy about this learning experience?

Click or tap here to enter text.

  1. Do you have any suggestions for how to improve the learning experience?

Click or tap here to enter text.





EVALUATION TECHNICAL ASSISTANCE
PRE–POST OUTCOME SURVEYS

On behalf of AmeriCorps (formerly known as the Corporation for National and Community Service), ICF is providing evaluation capacity building workshops throughout one year to grantees and sites that are participating in the [bundle name] evaluation. We are inviting you to complete this 15-minute survey at the beginning and end of your participation.

Participation in this survey is voluntary, but we hope you will participate because your experience and perspective are extremely valuable to assess the success of these workshops.

1. Awareness/Attitudes/Motivation


To what extent do you agree with the following statements in A and B? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

A. I think that an evaluation…

  1. Will help me understand my program.

  2. Will inform the decisions I make about my program.

  3. Will justify funding for my program.

  4. Will help to convince managers that changes are needed in my program.

  5. Is necessary to improve my program.

  6. Should involve program participants in the evaluation process.

  7. Will help improve services to target populations.

  8. Takes away resources that can be used to provide services.

  9. Is incorporated in my daily work.

B. I am motivated to…

  1. Learn about evaluation.

  2. Start evaluating my program.

  3. Support other staff to evaluate their program.

  4. Encourage others to buy into evaluating our program.





C. To what extent are the following factors important in your motivation to engage in

program evaluation? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

  1. A need to meet accountability requirements.

  2. A desire to enlighten and support government policymaking and planning.

  3. Changes in the organization (e.g., reorganization, new leadership, or vision).

  4. A mandate from leadership to increase the learning function of evaluation.

  5. A perceived lack of internal evaluation knowledge and skills.

  6. A desire to seek new or increased funding.

  7. A desire to use evaluation to make program improvements.

  8. Other factors (please specify): Click or tap here to enter text.

2. Skills/Knowledge/Behaviors

A. To what extent do you agree with the following statements? (Click “Choose an item”

to select the response that best corresponds to your answer for each one.)

I know how to…

    1. Recognize key concepts in evaluation (evidence, data, performance measurement, evaluation).

    2. Recognize the components of an evaluation plan.

    3. Describe what a “theory of change” is.

    4. Recognize how theory of change connects to a logic model.

    5. Recognize the attributes of a good evaluation question.

    6. Identify strategies to collect information from participants.

    7. Recognize features of process and outcome indicators of a program.

    8. Recognize who should collect data.

    9. Recognize when data should be collected.

    10. Identify common data analysis terms.

    11. Recognize how quantitative and qualitative analysis is performed.

    12. Recognize how to use the evaluation findings to improve a program.

    13. Read an evaluation report and recognize its basic components.

B. To what extent do you agree with the following statements? (Click “Choose an item”

to select the response that best corresponds to your answer for each one.)


In the past year, I helped my organization…

    1. Acquire funding to carry out an evaluation or hire an external evaluator (as part of a grant or through other means).

    2. Provide training or technical assistance to conduct evaluation (by staff, consultants, or other means).

    3. Conduct evaluations of programs funded by my organization.

    4. Use evaluation findings to improve programs funded by my organization.

    5. Use evaluation findings to demonstrate and communicate effectiveness of programs funded by my organization.

C. Does your organization have any of the following? (Click “Choose an item” to select

“yes” or “no” for each statement.)

    1. Staff position(s) or a group within your organization dedicated to evaluation.

    2. External evaluation partner(s)—consultant(s) or organization(s) that provide your organization with evaluation services.

    3. Part of the organization’s budget dedicated to evaluation.

3. Barriers

To what extent have you experienced the following barriers to conducting a program evaluation? (Click “Choose an item” to select the response that best corresponds to your answer for each one.)

    1. Not enough time.

    2. Not enough money.

    3. Not knowing how to conduct a program evaluation.

    4. Not having assistance with data collection.

    5. Not getting enough people to respond to surveys.

    6. Not knowing how to analyze data.

    7. Not having people to turn to for consultation and assistance.

    8. Not knowing what questions to ask.

    9. Not knowing how to write up results.

    10. Other factors (please specify): Click or tap here to enter text.

4. About the Respondent

  1. [Demographics] What race/ethnic group do you identify with? (Select one.)

☐ Asian/Asian-American

☐ Black/African-American

☐ Hispanic/Latino/Latina

☐ White

☐ Native American/American Indian/Alaskan Native/Native Hawaiian/
Pacific Islander

☐ Multi-racial or multi-ethnic (2+ races/ethnicities)

☐ Prefer not to respond

  1. [Demographics] What is your gender? (Select one.)

☐ Male

☐ Female

☐ Non-conforming

☐ Prefer not to respond

☐ Prefer to self-describe (please specify): Click or tap here to enter text.

  1. How long have you worked with your organization? (Select one.)

☐ Less than 1 year

☐ 2-5 years

☐ 6-10 years

☐ 11-20 years

☐ 20+ years





  1. What is your position in the organization? (Select one.)

☐ Leader (CEO, executive director)

☐ Other executive

☐ Manager/supervisor

☐ Staff

☐ Board member

☐ Other (please specify): Click or tap here to enter text.

  1. How long has your organization worked with AmeriCorps (formerly known as CNCS)? (Select one.)

☐ Less than 1 year

☐ 2-5 years

☐ 6-10 years

☐ 11-20 years

☐ 20+ years







File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorHolbrook, Brynn
File Modified0000-00-00
File Created2021-11-15
© 2024 OMB.report | Privacy Policy