Operation AmeriCorps Justification Part B_2016 1 4

Operation AmeriCorps Justification Part B_2016 1 4.docx

Operation AmeriCorps Evaluation

OMB: 3045-0176

Document [docx]

Download: docx | pdf

Operation AmeriCorps Evaluation

Shape1

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSIONS

B. Collections of Information Employing Statistical Methods

B1. Describe (including numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe for the Operation AmeriCorps national assessment consists of all Operation AmeriCorps grantees, defined as the legal applicants and sub-applicants listed on the application for federal funding. The respondent universe also includes grantees’ partner organizations involved in implementing the grant (as identified by the legal applicant and/or sub-applicant), and peripheral supporting organizations (as identified by key operating partners and the legal applicant and/or sub-applicant). There are an estimated 170 total respondents in this universe. We expect a 100% response rate to all data collection activities from grantees, as participation in the evaluation is required as a condition of the grant. We estimate a response rate of approximately 80-85% from key operating partners, and a 65-75% response rate from peripheral supporters, given the nature of their involvement in the grant.

We will also seek to interview small samples of AmeriCorps members working on an Operation AmeriCorps project, those members’ supervisors, and beneficiaries being served by the projects. These individuals will be identified by the project implementers, provided they are amenable to facilitating connections between the researchers and these individuals.

The maximum number of respondents in the study will be 340, as resources and logistics permit.

B2. Describe the procedures for the collection of information, including: Statistical methodology for stratification and sample selection; Estimation procedure; Degree of accuracy needed for the purpose described in the justification; Unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

For data collection activities targeting the grantee, a census approach will be used. For key operating partners and peripheral supporters, and for AmeriCorps members and project beneficiaries, sampling will be employed.

Purposeful samples of key operating partners and peripheral supporters will be selected for each project using two questions on the capacity survey (Questions 4 and 6), which asks the grantee to identify up to five key operating partners and up to three peripheral supporters. Although the primary purpose of this identification is to understand the network of organizations implementing the project, it will also serve to corroborate organizations targeted for interviews and focus groups. Convenience sampling will be used to select AmeriCorps members, supervisors, and beneficiaries willing and available to participate in a one hour focus group. Convenience sampling will also be used to identify project beneficiaries that would be willing and available to participate in interviews or focus groups.

This evaluation will employ both qualitative and quantitative data collection strategies to answer Research Questions 1-3; Research Questions 4 and 5 will be answered using qualitative data collection strategies. Each data collection strategy and its corresponding steps are described in detail below.

CNCS staff fall interviews: One-hour interviews with 11 federal agency staff members and 1 former staff member involved in the conceptualization and initial implementation of the Operation AmeriCorps grant program were conducted in October and November of 2015. The purpose of these interviews was to gather the perspectives of those closest to the realization of the grant program, and specifically to document how the idea for the program came about, the process by which it was operationalized, and how it was initially implemented. Staff from each AmeriCorps program office, grants management and monitoring, legal counsel, and the CEO’s office were selected to represent each phase of the idea formation and grant making process.

Grantee fall interviews: Initial interviews with representatives of each grantee were conducted in the fall of 2015, shortly after the beginning of the grant period, and were intended to orient the CNCS Office of Research and Evaluation (ORE) interviewers to each project’s basic operations, organizational structure, and intended outcomes. Specifically, researchers queried respondents on the purpose and design of the project; project management and infrastructure; relationships among partners; relationship with CNCS; and predicted opportunities and barriers to progress towards stated goals. These interviews were held between 10/15/15 and 10/27/15, lasted between 45 and 60 minutes, and were typically comprised of one grantee respondent and two to three ORE researchers¹.

CNCS will repeat these fall interviews in 2016, using the same interview protocol to produce a qualitative estimate of change over time in the areas noted above.

Grantee spring interviews: A series of spring interviews, which will form the basis for 10 individual case studies, will be conducted for all 10 Operation AmeriCorps projects. Up to five of these interview series will be conducted in person as part of a site visit, with the remainder conducted via phone. The interview series will be comprised of one-time, one hour interviews with up to nine individuals identified by the grantee: a representative from the organization tasked with the majority of the responsibility for implementing the project; the legal applicant and/or sub-applicant; one to three key operating partners; one to three peripheral supporters; and a member supervisor. These numbers may shift between interviewee types depending on the project and interviewee availability. No more than 8 interviews will be conducted per project.

Each interview will be conducted using an interview protocol tailored to each respondent group, aiming to document and assess grantees’ implementation of their Operation AmeriCorps projects (see attachment B). Respondents will be identified by consulting the legal applicant/sub-applicant and the main implementing organization for the relevant contacts in each respondent category. Specifically, the key operating partners will be primarily identified through Question 4 on the capacity survey; this will then be confirmed with the main implementing organization. Similarly, peripheral supporters will be identified via Question 6 on the capacity survey; this will also be confirmed with the main implementing organization. Member supervisors will ideally be knowledgeable staff members at the main implementing organization. They will be identified as potential respondents by the main implementing organization.

Once respondents are identified, interviews will be scheduled to take place between March and May, 2016. Interviews will occur one time for each respondent group, and will last no more than 60 minutes for each group. CNCS anticipates 100% participation rate from legal applicants, sub-applicants, and main implementing organizations, as cooperation with evaluation activities is required as part of the grant. CNCS anticipates potentially lower participation rates from key operating partners, peripheral supporters, and member supervisors given that they are not compelled to participate except by the encouragement of their partners. To achieve high participation rates, multiple attempts will be made to schedule interviews so that they occur at a time and in a mode that is convenient to the respondent. Additionally, ORE will emphasize the importance of the respondent’s role in the overall evaluation to increase likelihood of participation.

All interviews will be recorded using a tape recorder and subsequently will be transcribed verbatim. Audio files and interviewers’ written notes will be stored on CNCS’ secure server in a limited access project folder that is only accessible to members of the CNCS Office of Research and Evaluation. Upon completion of the evaluation in the late summer of 2017, all audio files and transcripts will be retained for a limited period of time, and then destroyed.

Grantee focus groups: To gather additional information for the 10 individual case studies, focus groups will be conducted with sets of AmeriCorps members participating in Operation AmeriCorps projects, as well as with sets of beneficiaries being served by the projects. The focus groups will occur one time with each respondent group (i.e. AmeriCorps members and project beneficiaries) per site, at sites for which focus groups are feasible². They will last no more than one hour. AmeriCorps member respondents will be chosen for participation by the member supervisor at each project site. Beneficiary respondents will be chosen for participation by a representative from the organization tasked with the majority of the responsibility for implementing the part of the project with the most interaction with beneficiaries³. No more than two focus groups will be conducted per project (one for members and one for beneficiaries), with no more than seven individuals (total) taking part in both focus groups. Each focus group will be conducted using a focus group protocol, tailored to each respondent group, aiming to document and assess respondents’ experiences with the Operation AmeriCorps project (see attachment C). No incentives will be given for participation. Up to five of these focus group sets may be conducted in person as part of a site visit, with the remainder conducted via phone.

Once respondents are identified, focus groups will be scheduled to take place between March and May, 2016. Since cooperation with evaluation activities is not required of the respondents targeted for focus groups, and since several sites operate in schools that will require advanced clearance for data collection activities to take place onsite, CNCS anticipates lower participation rates. To increase participation, multiple attempts will be made to schedule focus groups so that they occur at a time and in a mode that is convenient to the respondent. Additionally, ORE will work with a trusted point of contact familiar to the respondent group to both encourage participation and to emphasize the importance of the respondent’s role in the overall evaluation. Finally, ORE will work with the relevant point of contact at each site to secure any advance permission needed to conduct data collection activities (e.g. as would be needed at schools).

All focus groups will be recorded using a tape recorder and subsequently will be transcribed verbatim. Audio files and facilitators’ written notes will be stored on CNCS’ secure server in a limited access project folder that is only accessible to members of the CNCS Office of Research and Evaluation. Upon completion of the evaluation in the late summer of 2017, all audio files and transcripts will be retained for a limited period of time, and then destroyed.

Capacity survey: To establish baseline measurement of organizational capacity, and then track the change in that capacity over time, a survey will be administered to a representative from the main implementing organization (in other words, the organization responsible for implementing the core of the project’s activities) (attachment D). If deemed more appropriate, a representative from another key partner organization or from the legal applicant/sub-applicant organization may take the survey. The survey will be administered four times, twice per year of the grant (see Exhibit 1 for timing of administration). The survey will be administered online (via SurveyMonkey), and should take no more than 30 minutes to complete. A single individual will be identified as the survey respondent. ORE will work with the main implementing organization and the legal applicant/sub-applicant to identify the most appropriate individual to be the survey respondent. This individual should be someone who is closely involved in the administration of the grant and the project’s day-to-day activities, will be involved with the project through the life of the grant; and is empowered to speak on behalf of the project’s key partners.

Respondents will receive initial email communication from ORE containing a link to the survey (attachment E), as well as pre-notification from the Operation AmeriCorps program officer. The survey will remain open for three weeks after the initial email communication is distributed. Respondents will receive up to four email reminders from CNCS reminding them to complete the survey by the deadline. These email reminders will be sent approximately four, eight, and 12 days after the initial email communication is sent; a fourth and final reminder will be sent shortly before last day of the survey open period. ORE will monitor completion rates during the survey open period and if necessary will proactively engage the Operation AmeriCorps program officer to encourage survey completion. After the deadline has passed, ORE will work directly with the Operation AmeriCorps program officer to ensure all sites have completed the survey; survey respondents will be contacted either via email or phone as needed to complete the online survey. CNCS anticipates 100% response rate, as cooperation with evaluation activities is required of the targeted survey respondents as part of the grant. To ensure high participation rates, ORE notified grantees of all planned data collection activities, including the capacity survey, at a gathering of AmeriCorps grantees in September, 2015. Additionally, ORE will collaborate with the Operation AmeriCorps program officer to pre-notify respondents of the survey open period, and will send multiple email reminders to respondents once the survey is live. If necessary, follow up contact will ensure that all respondents complete the survey.

Completed survey data will be stored in the SurveyMonkey online system. Individual respondents’ personally identifiable information will not be collected on the SurveyMonkey platform. ORE will download the data to a secured project folder once all respondents have completed the survey at the conclusion of each wave of survey administration.

Document review: In addition to the data collection activities described above, ORE will request additional grantee administrative data and documents to conduct a document review. Documents of interest include, but are not limited to, the application for federal funding, program logic model, VISTA Assignment Descriptions (VADs), and GPR and performance measures submissions. Additional performance or evaluation data are also of interest. Document requests will be made no more than twice per year directly to grantees as well as through the Operation AmeriCorps program officer.

Data storage

All audio and electronic documents related to the evaluation will be stored on CNCS’ secure server in a project folder that is only accessible to members of the Office of Research and Evaluation. This includes interview and focus group notes; transcripts; audio files from interviews and focus groups; survey data files; and additional data and documents sent by grantees. Upon completion of the evaluation in the late summer of 2017, all audio files and transcripts will be retained for a limited period of time, and then destroyed.

Exhibit 1. Data collection timeline

Quarter	Month	Data collection activity

Fall 2015	Sept	Application review
	Oct	Grantee interviews; CNCS staff interviews; administrative document request
	Nov
Winter 2015/2016	Dec
	Jan
	Feb	Survey
Spring 2016	Mar	Grantee, main implementing organization, and partner interviews and focus groups; selected site visits
	Apr	(March activities continue as needed); administrative data collected (GPR, member activity, other outcomes data); administrative document request
	May	(March activities continue as needed)
Summer 2016	Jun
	Jul
	Aug	Administrative data collected (GPR, member activity, other outcomes data); administrative document request; Survey
Fall 2016	Sept	Continuation application review (if necessary)
	Oct	Grantee interviews; CNCS staff interviews; administrative document request
	Nov
Winter 2016/2017	Dec
	Jan
	Feb	Survey
Spring 2017	Mar	Grantee, main implementing organization, and partner interviews and focus groups; selected site visits
	Apr	(March activities continue as needed); administrative data collected (GPR, member activity, other outcomes data); administrative document request
	May
Summer 2017	Jun
	Jul
	Aug	Administrative data collected (GPR, member activity, other outcomes data); Survey

Analysis of qualitative data

Qualitative data generated by the interviews and focus groups described above will be analyzed using NVivo qualitative analytic software, using the project research questions as an analytic framework. Transcripts will be coded, and queries will be run to identify themes and key patterns relevant to the two categories of research questions outlined in Justification Part A (i.e. internally focused questions related to the grant conceptualization and implementation process, and externally focused questions related to project implementation and preliminary achievement of outcomes). ORE will compare patterns, similarities, and differences between grantees and within like groups of grantees (groups may be constructed based on common outcome, funding priority area, or intervention type). When possible, triangulation will be used to confirm patterns and themes detected by the analysis. This process will also be applied to the secondary data collected by the document requests. ORE may construct tables, graphics, or other visuals to help clarify relationships, patterns, or trends identified in the data.

Analysis of quantitative data

Quantitative survey data will be exported as .csv files from SurveyMonkey and then will be uploaded into SPSS or STATA for analysis. ORE will run descriptive statistics (e.g. frequencies and crosstabs) to compose a profile of organizational capacity for each project site. A complete record of site demographics (e.g. grant size, number of partners, geographic location, etc.) as well as response rates and details related to survey administration will be maintained for each site. These additional variables may be used to conduct more complex analyses. Quantitative data resulting from GPR reporting will be compared to stated targets to determine whether or not targets were met. CNCS anticipates that any additional quantitative data submitted as part of the administrative data and document requests (i.e. additional performance measurement and outcomes data) may lack sufficient documentation to conduct statistical analyses, and therefore will be reviewed in a similar manner to qualitative data.

Non-response analysis

Since participation in the evaluation is required as part of the grant, and since all survey questions will be required, we do not anticipate item-specific nonresponse.

Reporting

An interim report(s) containing preliminary analysis of staff interviews and fall 2015 grantee interviews will be completed in spring of 2016 and will be shared with agency leadership. One or more written reports will be distributed at the end of the first year to senior agency leadership, internal staff that participated in staff interviews, and Operation AmeriCorps grantee organizations. This report will also contain suggestions and recommendations for improvements to the Operation AmeriCorps grant-making process that could be implemented in real time for the benefit of future cohorts or continuing grantees.

Final reports will be produced at the end of the grant period in late fall of 2017.

The final evaluation report will be distributed to grantees and agency leadership/staff, and will be published on nationalservice.gov for public consumption.

B3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

To achieve high participation rates for the spring interviews and focus groups, particularly for respondent groups that are not compelled to participate in the evaluation, ORE will emphasize the importance of gathering and incorporating the perspectives of these groups into the evaluation of the initiative. This language will be incorporated into interview outreach and reminder communications, and will also be communicated by the implementing organization (in order to leverage their relationship with these respondent groups).

To improve response rates on the capacity survey, four reminders will be provided via email at intervals of four, eight, and 12 days post initial email communication containing the survey link; a fourth and final reminder will be sent shortly before the last day of the survey open period. CNCS expects 100% response to the survey and will work with the grantee, implementing organization, and the Operation AmeriCorps program officer to achieve complete participation.

B4. Describe any tests of procedures or methods to be undertaken.

Refer to Justification Package A, section A.1, pages three to four.

B5. Provide the name and telephone number of individuals consulted on statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

CNCS staff will oversee data collection activities and analyze the data collected. The individuals at CNCS from the Office of Research and Evaluation assigned to this project include:

Diana Epstein, Ph.D, Senior Research Analyst, 202-606-7564
Joseph Breems, MSPPM, Policy Analyst, 202-606-6992
Adrienne DiTommaso, MPA, Research Analyst, 202-606-3611

References

McKinsey Organizational Capacity Assessment Tool. Accessed July, 2015. http://mckinseyonsociety.com/ocat/.

Attachments

Grantee Fall Interview Protocol
Grantee Spring Interview Protocols
Grantee Focus Group Protocols
Capacity Survey
Capacity Survey Communications
2015 Fall Grantee Interview Clearance

Grantee Fall Interview Protocol

FALL 2016 INTERVIEW INSTRUMENT FOR OPERATION AMERICORPS GRANTEE

TARGETED RESPONDENT: These interview questions are intended for Operation AmeriCorps grantees. The interview will be administered to the grantee-designated representative of the grant contact organization.

CONTENT: This document contains interview questions to be asked of grantees as well as reminders of previous document requests as needed.

Sample call introduction (assuming prior communication to set up the interview):

Hello. This is (name) and I am with the Office of Research and Evaluation at the Corporation of National and Community Service (CNCS). I’m calling to talk with you about your experiences thus far with the Operation AmeriCorps grant process and learn more about the project for which the grant was awarded. Is this still a good time to talk? [If not, reschedule and note new date/time]. Great! Before we get started, let’s go over a few quick points before we get started. This is part of the national evaluation of Operation AmeriCorps, which is being conducted by me and my colleagues in the Office of Research and Evaluation at CNCS. [If applicable: I also have my colleague [NAME] on the phone with me today to take notes and help with some of the questions.]

Our interview today will last about an hour, and we’ll mainly be covering questions about the structure and capacity of your program, as well as your experience with the grant making process. We really appreciate candor, so feel free to be as honest or blunt as you want.

I’d like to start off by asking for your permission to conduct this interview. All information (audio and written) is confidential and interview responses will be stored in a secure location that is only accessible to members on the study team. Only study team members will see individual responses that can be linked to you and all personally identifying information will be redacted from the transcripts of this interview. Data files and study reports will be shared with CNCS only after study team members have examined the data to be free of information that directly identifies you. Additionally, disbursement of the current Operation AmeriCorps grant is in no way tied to what is discussed in this interview. With this in mind, are you ok moving forward with the interview?

____Check when gives verbal permission and enter time/date ____

I’d also like your permission to audio record our conversation solely for the purpose of note-taking. All information (audio and written) is confidential. Is this okay?

___ Check if gives verbal permission and enter time/date ____

We recognize each grantee is in a different place with regard to Operation AmeriCorps and the implementation of your project so if there are questions that you are not familiar with or that you aren’t sure of the answer to, it’s not a problem. Also, if you think of anything after our conversation today, you are welcomed to email us with additional thoughts or helpful documents.

Do you have any questions before we begin?

How did you hear about Operation AmeriCorps? Can you tell me about how the decision to apply was made? Prompts: This would include who was involved in that decision, what was the motivation, the process for making the decision, and why Operation AmeriCorps was considered the best choice. If you were not directly involved in the decision, can you tell me about your understanding of how it was made?

What is your Operation AmeriCorps project looking to accomplish? How will you know if the project has accomplished its goals? Prompts: Can you walk us through the outcomes in your logic model? What are the short term goals? What are long term goals? Ask for documents outlining the strategy and structure of the project, such as a logic model or mission and vision statements, if the organization has not already sent.

Could you describe the activities that your organization carries out for your Operation AmeriCorps project? Prompts: By this we mean what role your organization plays in carrying out this program. Can you explain in more detail? What is your specific role in this work? As a note for interviewer – typical “backbone” organization is involved in overseeing the coordination of activities, managing communications and other information-sharing, convening partners and supporters as needed, and dedicating staff to this administration work.

“Operating partners” are the individuals, groups, and organizations carrying out the activities of this project, involved in the decision-making, communicating regularly with other partners, and sharing strategy, information, and resources to make this project happen. Who would you call “operating partners” for the Operation AmeriCorps project? What formal agreements (written documentation) are there, if any, among and between operating partners? Prompts: Are there any others who are part of the day-to-day of this program that haven’t been named yet? Ask whether there is documentation of partners available if the organization has not already sent.

What brought this group of operating partners together? When did this take place?

Which components of the Operation AmeriCorps project are different partners responsible for? Tell me more about how organizations’ roles were decided for this program. How are these different responsibilities coordinated among partners? Prompts: Please describe these responsibilities in further detail. Ask whether there is documentation of the working relationships among partners, such as an organizational chart or written agreements, if the organization has not already sent.

How are communications and decision-making managed among operating partners? What systems are in place to support this management? This would include data systems and other information-sharing platforms as well as established processes like a newsletter or regular meetings.

“Peripheral supporters” are the individuals, groups, and organizations occasionally receiving information about and advising on the direction of the project, advocating for the project either to community members or influential leaders, and potentially providing financial or other resource support. Who would you call “peripheral supporters” of this project? What formal agreements (written documentation) are there, if any, between project partners and supporters? Prompts: If there are no formal agreements, how was an understanding created for this relationship? Ask whether there is documentation of supporters and the agreements between them and project partners if the organization has not already sent.

What CNCS programs, if any, have you been a grantee of prior to being awarded the Operation AmeriCorps grant? To your knowledge, have any operating partners been AmeriCorps grantees prior to this? If not a past grantee, move to Question 10. If a past grantee: How does being a grantee through Operation AmeriCorps compare to being a grantee in years past? Prompts: What was different about the grant-making process? What do you perceive to be different about program operations?

In general, how easy or complicated did you find the Operation AmeriCorps application process to be? How many CNCS points of contact did you have? Now that you are managing your grant, how many CNCS points of contact do you have? How would you compare this process with other non-CNCS grants you’ve applied for? Prompts: If interviewee was not part of the application process, ask about what they know of the experience of those who were.

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

What challenges have you encountered in the beginning months of your Operation AmeriCorps project? How are you trying to overcome them? Prompts: Any critical unforeseen challenges? Tell me about how recruitment of Corps members and volunteers has been affected.

What do you think will be your greatest challenges or barriers to achieving the goals of your Operation AmeriCorps project? What factors do you think will be critical to achieving success?

Grantee Spring Interview Protocols

Operation AmeriCorps National Evaluation

Spring Interviews- Protocol and Questions

Document 1: “Main Implementing” organization

Respondent Name:

OPAC grantee:

Interviewer Names:

Date:

Duration of call:

Follow up items:

Hi [respondent name], this is [interviewer name/s] calling from the Corporation for National and Community Service, how are you?

Is this still a good time for you to talk about [insert topic]? [If no, reschedule a time on the phone]. [If yes:] Great!

Let’s cover a few quick points before we get started with some questions. First, we want to thank you for taking the time to speak with us today about your Operation AmeriCorps project; our discussion today is going to help get a better sense of how things are unfolding for our grantees as they move into the latter halves of the program year. Second, we want to remind you that your responses to the questions we’ll ask you today are confidential, and are only going to be used by members of the evaluation team for the purpose of the national evaluation of Operation AmeriCorps. Your data is being protected and you won’t be personally identified or linked to your responses. Notes from the interview today will be stored electronically on a secure server, and at the end of the evaluation, we will destroy these records. Relatedly, we’ll be recording today’s interview to make sure we don’t miss anything important. The recordings will be transcribed; you will not be identified, nor will your responses be directly linked to your name. The audio files from your interview will be stored on a secure server, and only researchers from our evaluation team will have access to these. We will keep the files on the server until the evaluation is completed, and at that point they’ll be destroyed. Do we have your permission to record the interview? Finally, we want you to know that you can be as honest or blunt as you want; we really want to make sure that we’re capturing the realities of running an Operation AmeriCorps program, so there is no need to hold back on your answers and we encourage you to be candid. Do you agree to participate today?

Ok, the way this will work is that we have a short list of questions, that cover [insert topics: capacity your program is building; the different service streams your program is using, and how they’re interacting; early evidence of outcomes; and what you think about implementation of the Operation AmeriCorps grant]. [Interviewer may note that the question topics were pre-sent]. Also, please feel free at the end to add anything you think we missed that you think is important.

Q1: Interaction of Service Streams Questions:

We’d like to start out by hearing about what your program has been up to for the past few months.

Can you catch us up on what is currently going on in the project?
Have you made any mid-year changes to the project’s objectives?
- Do you think the mission or goals of the project have changed? If so, how and why? Do you think they will need to change in the future?
- What about the logic model, and theory of change? [Interviewers: a theory of change is a program’s statement of the problem it is addressing, the outcomes it hopes to achieve, and how it proposes to achieve them; a logic model is a visual representation of a program’s theory of change.]

Let’s talk a little more specifically about member activities for each service stream. [Interviewers: you may need to note which service streams they’re using in case you need to prompt them].

Can you describe what members from each service stream are doing as part of your Operation AmeriCorps project? [Interviewers: if respondent does not understand “service stream”, prompt with “type of AmeriCorps program”, or list the service streams you noted at the beginning of the interview].
Can you tell us about how members are being managed and supervised?
Have you had to change any of the roles, responsibilities, or activities of members since you were planning the project or since it started in the fall?
- If so, why?
Tell us about how national service members and volunteers interact in their work.

Q2: Community Capacity Questions:

As you’re aware, one of the main objectives of this grant is to build community capacity. You might be familiar with the term “community capacity”, which refers to the capabilities or competencies a group or network of organizations has to address a community problem. We want to talk a little about the community capacity you and your partners are building to successfully implement this project, achieve your goals, and sustain this work into the future.

Can you describe how your program’s partnership structure has evolved in the last several months?
- Have these changes been made formally or informally?
- Have the roles and responsibilities of each player changed?
- What additional resources have you been able to recruit to the program? Were there any you were planning on that didn’t materialize?
Can you describe for me the conversations you and your partners have had about sustaining this program long-term?

[Interviewers: if approaching the 60 minute mark of the interview and you do not believe you will be able to finish, set up an additional interview to finish Q3 and Q4.]

Q3: Early Evidence of Outcomes:

We’re going to move on to early evidence for outcomes. We realize that many Operation AmeriCorps programs are new, and that one year is really not enough time to see a lot of progress on the outcomes you’re working towards. Regardless, we wanted to ask you about what kind of preliminary progress you’re seeing towards achieving your program’s outcomes.

What kind of early progress are you seeing towards achieving outcomes?
Is there anything surprising or interesting you are seeing as the program works towards preliminary outcomes?
What kinds of challenges or barriers have you had to overcome?
- [If the respondent mentions an unexpected lack of progress] How have you dealt with unexpected lack of progress?

Q4: Grant Administration and Implementation:

Now we want to completely switch subjects and talk about the grant administration and implementation process. The Operation AmeriCorps grants were set up in part to reduce the burden on grantees who want to apply for members from multiple streams of national service, which normally would require coordinating multiple grants and their requirements.

Can you start by telling us about how you’ve been handling grant administration and implementation?
Can you tell us about how the combination of multiple programs into one grant has affected how you administer or implement the grant?
- What has communication been like with CNCS?
  - [If respondent mentions particular challenges or successes, probe further]
- Can you describe your approach to managing and reporting on funds?
- What have you seen as challenges in managing this grant?
- What do you see as benefits in managing this grant?

[After finishing question list or reaching end of scheduled time] Thank you so much for speaking with us, this has been really insightful. If you think of anything in the coming days that you want to tell us about, please feel free to email me or [note taker’s name] directly. Thanks so much for your time and your service!

Operation AmeriCorps National Evaluation

Spring Interviews- Protocol and Questions

Document 2: Legal applicant organization

Respondent Name:

OPAC grantee:

Interviewer Names:

Date:

Duration of call:

Follow up items:

Hi [respondent name], this is [interviewer name/s] calling from the Corporation for National and Community Service, how are you?

Is this still a good time for you to talk about [insert topic]? [If no, reschedule a time on the phone]. [If yes:] Great!

Ok, the way this will work is that we have a short list of questions, that cover [insert topics: capacity your program is building; the different service streams your program is using, and how they’re interacting; and what you think about implementation of the Operation AmeriCorps grant]. Also, please feel free at the end to add anything you think we missed that you think is important.

[Interviewers: the following questions should only be asked of organizations that are the legal applicant/sub-applicant but that are NOT also the backbone organization.]

Q1: Interaction of Service Streams Questions:

We’d like to start out by hearing about what your organization has been up to for the past few months.

Can you catch us up on what is currently going on in the project?
Have you made any mid-year changes to the project’s objectives?
- Do you think the mission or goals of the project have changed? If so, how and why? Do you think they will need to change in the future?
- What about the logic model, and theory of change? [Interviewers: if the respondent is confused about theories of change and logic models, you can use the following definitions: a theory of change is a program’s statement of the problem it is addressing, the outcomes it hopes to achieve, and how it proposes to achieve them; a logic model is a visual representation of a program’s theory of change.]

Let’s talk a little more specifically about some of the activities taking place. [Interviewers: you may need to note which service streams they’re using in case you need to prompt them].

Are you working directly with members?
- [If yes] Can you describe what members from each service stream are doing as part of your Operation AmeriCorps project? [Interviewers: if respondent does not understand “service stream”, prompt with “type of AmeriCorps program”, or list the service streams you noted at the beginning of the interview].
  - How are members tracking and reporting their activities?
- Can you tell us about how members are being managed and supervised?
- Have you had to change any of the roles, responsibilities, or activities of members since you were planning the project or since it started in the fall?
  - If so, why?
- Tell us about how national service members and volunteers interact in their work.
[If no] What is your primary role or responsibility when it comes to the project’s activities?
- Has this changed since the fall?
  - If so, why?

Q2: Community Capacity Questions:

Can you describe how your program’s partnership structure has evolved in the last several months?
- Have these changes been made formally or informally?
- Have the roles and responsibilities of each player changed?
- What additional resources have you been able to recruit to the program? Were there any you were planning on that didn’t materialize?
Can you describe for me the conversations you and your partners have had about sustaining this program long-term?

Q3: Early Evidence of Outcomes:

What kind of early progress are you seeing towards achieving outcomes?
Is there anything surprising or interesting you are seeing as the program works towards preliminary outcomes?
What kinds of challenges or barriers have you had to overcome?
- [If the respondent mentions an unexpected lack of progress] How have you dealt with unexpected lack of progress?

Q4: Grant Administration and Implementation:

Now we want to completely switch subjects and talk about the grant administration and implementation process.

Can you start by telling us about how you’ve been handling grant administration and implementation?
- Have you had an AmeriCorps grant in the past?
  - [If yes] How similar is the administration and implementation process to what you’re used to? How is it different?
- Have you had a grant similar to the Operation AmeriCorps grant before? [Interviewers: some examples might include place-based initiative grants, community change grants, etc.]

The Operation AmeriCorps grants were set up in part to reduce the burden on grantees who want to apply for members from multiple streams of national service, which normally would require coordinating multiple grants and their requirements.

Can you tell us about how the combination of multiple programs into one grant has affected how you administer or implement the grant?
- What has communication been like with CNCS?
  - [If respondent mentions particular challenges or successes, probe further]
- Can you describe your approach to managing and reporting on funds?
- What have you seen as challenges in managing this grant?
- What do you see as benefits in managing this grant?

Operation AmeriCorps National Evaluation

Spring Interviews- Protocol and Questions

Document 3: Select Operating Partners

Respondent Name:

OPAC grantee:

Interviewer Names:

Date:

Duration of call:

Follow up items:

Hi [respondent name], this is [interviewer name/s] calling from the Corporation for National and Community Service, how are you?

Is this still a good time for you to talk about [insert topic]? [If no, reschedule a time on the phone]. [If yes:] Great!

Ok, the way this will work is that we have a short list of questions, that cover [insert topics: capacity your program is building; the different service streams your program is using, and how they’re interacting; and what you think about implementation of the Operation AmeriCorps grant]. Also, please feel free at the end to add anything you think we missed that you think is important.

Q1: Interaction of Service Streams Questions:

Can you start by telling us about your organization’s role in the Operation AmeriCorps project?
- Do you work with members?

If yes: Can you describe what members are currently doing on the project?
- How are you supervising and managing members?
- How are the different streams of member activities complementing one another?
- Have there been challenges to coordinating the members’ activities?
  - How are you working to overcome these challenges?
  - How has the implementation of member activities differed from what you planned?
- How have you been tracking member activities?
If no: does your organization interact with members at all?

Are you in charge of providing financial resources?
- If yes: can you describe these resources and their role in the project? [Interviewers: probe for details if they are comfortable sharing, such as type, amount, and source]
Are you in charge of providing non-financial resources?
- If yes: can you describe these resources and their role in the project? [Interviewers: probe for details if they are comfortable sharing, such as type, amount, and source]

Q2: Community Capacity Questions:

Can you describe the program’s partnership structure?
- Who are some of your key partners?
- What types of partners do you believe add significant value to the project? Do you feel the project lacks any of these?
Do you believe the partnership structure has evolved in the last several months?
- Have these changes been made formally or informally?
- Has your organization’s role or responsibilities changed?
- Do you think the mission or goals of the project have changed?
Have you made any mid-year changes to your organization’s objectives in the Operation AmeriCorps project? If so, please describe these changes.
Can you describe for me the conversations you and your partners have had about sustaining this program long-term?

Q3: Preliminary Evidence of Effectiveness:

We’d now like to ask you about your perception of preliminary progress towards achieving the project’s outcomes.

What kind of early progress are you seeing towards achieving outcomes?
- Is there anything surprising or interesting you are seeing as the program works towards preliminary outcomes?
What kinds of challenges or barriers have you had to overcome?
- [If the respondent mentions an unexpected lack of progress] How have you dealt with unexpected lack of progress?

Q4: Grant Administration and Implementation:

Now we want to switch subjects and talk about the grant administration and implementation process.

Are you responsible for any part of the grant administration process?
[If yes]: How has your organization been involved in the administrative aspects of implementing the grant? This might include involvement in tracking and reporting on member activities, coordinating funding, or working directly with CNCS or other funders to ensure the grant is in compliance.
- Does your role require you to be in direct contact with CNCS? If so, how have you found the process of communicating with CNCS for this grant?
  - [If respondent mentions particular challenges or successes, probe further]
- [If respondent has indicated that they manage CNCS funds]: Can you describe your approach to managing and reporting on funds?
What have you seen as challenges in participating in this grant?
What do you see as benefits in participating in this grant?

Operation AmeriCorps National Evaluation

Spring Interviews- Protocol and Questions

Document 4: Select peripheral supporters

Respondent Name:

OPAC grantee:

Interviewer Names:

Date:

Duration of call:

Follow up items:

Hi [respondent name], this is [interviewer name/s] calling from the Corporation for National and Community Service, how are you?

Is this still a good time for you to talk about [insert topic]? [If no, reschedule a time on the phone]. [If yes:] Great!

Ok, the way this will work is that we have a short list of questions, that cover [insert topics: capacity your program is building; the different service streams your program is using, and how they’re interacting; and what you think about implementation of the Operation AmeriCorps grant]. Also, please feel free at the end to add anything you think we missed that you think is important.

Q1: Interaction of Service Streams Questions:

Can you start by telling us about your organization’s role in the Operation AmeriCorps project?
- Do you work with members?

If yes: Can you describe what members are currently doing on the project?
- How are you supervising and managing members?
- How are the different streams of member activities complementing one another?
- Have there been challenges to coordinating the members’ activities?
  - How are you working to overcome these challenges?
  - How has the implementation of member activities differed from what you planned?
- How have you been tracking member activities?
If no: does your organization interact with members at all?

Are you in charge of providing financial resources?
- If yes: can you describe these resources and their role in the project? [Interviewers: probe for details if they are comfortable sharing, such as type, amount, and source]
Are you in charge of providing non-financial resources?
- If yes: can you describe these resources and their role in the project? [Interviewers: probe for details if they are comfortable sharing, such as type, amount, and source]

Q2: Community Capacity Questions:

Can you describe the project’s partnership structure?
- Who are some of your key partners?
- What types of partners do you believe add significant value to the project? Do you feel the project lacks any of these?
Do you believe the partnership structure has evolved in the last several months?
- Have these changes been made formally or informally?
- Has your organization’s role or responsibilities changed?
- Do you think the mission or goals of the project have changed?
Have you made any mid-year changes to your organization’s objectives in the Operation AmeriCorps project? If so, please describe these changes.
Can you describe for me the conversations you have had with the program about long-term sustainability?

Q3: Preliminary Evidence of Effectiveness:

We’d now like to ask you about your perception of preliminary progress towards achieving the project’s outcomes.

What kind of early progress are you seeing towards achieving outcomes?
- Is there anything surprising or interesting you are seeing as the program works towards preliminary outcomes?
What kinds of challenges or barriers have you had to overcome?
- [If the respondent mentions an unexpected lack of progress] How have you dealt with unexpected lack of progress?

Q4: Grant Administration and Implementation:

Are you responsible for any part of the grant administration process?
- [If yes] Can you describe your role and responsibilities?

Operation AmeriCorps National Evaluation

Spring Interviews- Protocol and Questions

Document 5: Select supervisors of national service members, volunteers

Respondent Name:

OPAC grantee:

Interviewer Names:

Date:

Duration of call:

Follow up items:

Hi [respondent name], this is [interviewer name/s] calling from the Corporation for National and Community Service, how are you?

Is this still a good time for you to talk about [insert topic]? [If no, reschedule a time on the phone]. [If yes:] Great!

Ok, the way this will work is that we have a short list of questions, that cover [insert topics: capacity your program is building; the different service streams your program is using, and how they’re interacting; and what you think about implementation of the Operation AmeriCorps grant]. Also, please feel free at the end to add anything you think we missed that you think is important.

Q1: Interaction of Service Streams Questions: What partners do you interact with? Describe your role.

Can you start by telling us about your role in the Operation AmeriCorps project?
- What are your day to day responsibilities?
- Are you in charge of managing any financial resources?
  - If yes: can you describe your responsibilities and tasks?
- Are you in charge of managing any non-financial resources?
  - If yes: can you describe your responsibilities and tasks?
- Have there been challenges to coordinating the members’ activities?
  - How are you working to overcome these challenges?
  - How has the implementation of member activities differed from what you planned?
- How have you been tracking member activities?
Do you believe the different streams of member activities are complementing one another? If so, how?

Q2: Community Capacity Questions:

What is your understanding of the goals of this project?
Who are some of your key partners in your day to day work?

Q3: Preliminary Evidence of Effectiveness:

We’d now like to ask you about your perception of preliminary progress towards achieving the project’s outcomes.

What kind of early progress are you seeing towards achieving outcomes?
- Is there anything surprising or interesting you are seeing as the program works towards preliminary outcomes?
What kinds of challenges or barriers have you had to overcome?
- [If the respondent mentions an unexpected lack of progress] How have you dealt with unexpected lack of progress?

Q4: Grant Administration and Implementation:

Are you responsible for any part of the grant administration process?

Grantee Focus Group Protocols

Operation AmeriCorps National Evaluation

Spring Focus Groups- Protocol and Questions

Document 1: National service members and volunteers

Respondent Names:

OPAC grantee:

Facilitator Names:

Date:

Duration of session:

Follow up items:

Hi everyone, my name is [facilitator names] from the Corporation for National and Community Service.

Let’s cover a few quick points before we get started with some questions. First, we want to thank you for taking the time to speak with us today about the Operation AmeriCorps project you are a part of; our discussion today is going to help us get a better sense of how things are unfolding for the project. Second, we want to remind you that your responses to the questions we’ll ask you today are confidential, and are only going to be used for the purpose of the national evaluation of Operation AmeriCorps. Your data is being protected and you won’t be personally identified or linked to your responses. We also ask that you do not share the responses of any other focus group members outside of this room.

We’ll be recording today’s interview to make sure we don’t miss anything important. The recordings will be transcribed, and then your identifying information will be removed. Do we have your permission to record the conversation? Finally, we want you to know that you can be as honest or blunt as you want; we really want to make sure that we’re capturing the realities of how the Operation AmeriCorps program is running, so there is no need to hold back on your answers and we encourage you to be honest. Do you agree to participate today?

Ok, the way this will work is that we have a short list of questions, that cover capacity your program is building; the different service streams your program is using, and how they’re interacting; and early evidence of outcomes. That said, please feel free to bring up thoughts and reactions you feel are relevant to the conversation at any time, as well as respond to what other focus group members are saying. This is intended to be an informal group conversation, not a rigid, one-on-one interview.

Q1: Interaction of Service Streams Questions:

We want to start off by talking about national service member and volunteer activities. We note that [insert known CNCS streams of service and other volunteer types] are involved in this project. Are there any others that need to be included?

When did you start serving as part of this project?
Can you describe what you and other members from your CNCS program (such as AmeriCorps, VISTA, NCCC, or Senior Corps) or volunteer group are doing as part of the Operation AmeriCorps project?
Can you tell us about how you are being managed and supervised?
Have you had your role, responsibilities, or activities changed since you started in the fall? If so, how have they changed?
Tell us about how you interact with other national service members and volunteers in your service.

Q2: Community Capacity Questions:

We want to talk a little about your understanding of what this Operation AmeriCorps project is trying to accomplish and how it’s being implemented.

How would you describe the goals of this Operation AmeriCorps project?
- How were these goals first communicated to you? Have they changed at all since you began serving?
- What part do you see your service playing in reaching these goals?
- Do you feel all national service members and volunteers on this project have a clear understanding of its goals? Why or why not?
What organization do you see as the primary coordinator or administrator of this project?
- What does this organization do that makes you see them as the primary coordinator?
- What different organizations or community groups do you interact with regularly in your service?

Q3: Early Evidence of Outcomes:

We’re going to move on to early evidence for outcomes. We realize that many Operation AmeriCorps projects are new, and that one year is really not enough time to see a lot of progress on the outcomes you’re working towards. Regardless, we wanted to ask you about what kind of preliminary progress you’re seeing towards achieving the project’s outcomes.

Do you feel the project’s desired outcomes are being achieved? Can you provide examples of why you feel this way?
Is there anything that you feel is working out better than you expected? Why do you think so?
What sort of barriers or challenges are you experiencing?

Operation AmeriCorps National Evaluation

Spring Focus Groups- Protocol and Questions

Document 2: Project beneficiaries

Respondent Names:

OPAC grantee:

Facilitator Names:

Date:

Duration of session:

Follow up items:

Hi everyone, my name is [facilitator names] from the Corporation for National and Community Service.

Let’s cover a few quick points before we get started with some questions. First, we want to thank you for taking the time to speak with us today about your experience with the Operation AmeriCorps project [insert program name if different]; our discussion today is going to help us get a better sense of how things are unfolding for the project. Second, we want to remind you that your responses to the questions we’ll ask you today are confidential, and are only going to be used for the purpose of the national evaluation of Operation AmeriCorps. Your data is being protected and you won’t be personally identified or linked to your responses. We also ask that you do not share the responses of any other focus group members outside of this room.

Ok, the way this will work is that we have a short list of questions, that cover the kinds of activities you’ve taken part in, the AmeriCorps members and volunteers who have helped you, and if you’ve seen benefits from being in the program. That said, please feel free to bring up thoughts and reactions you feel are relevant to the conversation at any time, as well as respond to what other focus group members are saying. This is intended to be an informal group conversation, not a rigid, one-on-one interview.

Q1: Interaction of Service Streams Questions:

We want to start off by talking about you’re the activities you’ve taken part in and your interactions with AmeriCorps members and other volunteers.

Tell us about the activities you take part in as part of this program.
Can you tell us about interacting with AmeriCorps members and other volunteers?
When did you start taking part in this program?

Q3: Early Evidence of Outcomes:

We’re going to move on to early evidence for outcomes. In other words, we want to understand if you’re seeing any positive changes or benefit because of this program.

What do you think this program is trying to achieve (what are its goals)?
Why did you decide to take part in it?
Do you feel taking part in this program is helping you? Why or why not? Please provide examples of changes you’ve seen in as a result of this program.
What do you find particularly helpful about this program?
What do you feel is missing that could make the program better for you?

[After finishing question list or reaching end of scheduled time] Thank you so much for speaking with us, this has been really helpful. If you think of anything in the coming days that you want to tell us about, please feel free to email me or [note taker’s name] directly. Thanks so much for your time!

Capacity Survey

Capacity Assessment Grid: Survey Supplement for OPAC Grantees

Beyond providing a critical mass of national service members to support transformational community change, Operation AmeriCorps seeks to build the organizational capacity of participating non-profits. Organizational capacity is your organization’s ability to develop, operate, and sustain programmatic activity. The following survey questions ask about a few areas of organizational capacity. Your answers to these questions will not be shared outside of this evaluation, and you will not be identified by your responses. The survey will take approximately 10 minutes to complete.

Please indicate whether the following statements are true or not by selecting “Yes” or “No” for each question, or by selecting the appropriate response, and completing the response as requested.

[Mission] Our Operation AmeriCorps project has a written expression of the project’s reason for existence, such as a mission statement. [Y/N]
[Vision] All organizations involved in our Operation AmeriCorps project have a shared understanding of what success looks like for this project. Select the response you feel comes closest to the truth: [Not at all true, somewhat true, mostly true, completely true]
[Overarching goals] The project’s vision has been translated into a concrete set of measurable goals. [Y/N]

We would now like to ask you some questions about the partnerships established for your project. The first set of questions focuses on the project’s operating partners. Operating partners can be thought of as those individuals, groups, and organizations involved in carrying out the day-to-day activities of the project. These partners can range from non-profit organizations and foundations to private businesses to schools and government agencies, to name a few.

Before responding to the questions below, please list the names of up to 5 Operation AmeriCorps project operating partners who are necessary to carrying out the project:

[Operating Partners] For the project’s operating partners as a group, to what extent are the following statements true?:

4a. Roles and responsibilities among operating partners are clearly defined.

[Not at all true, somewhat true, mostly true, completely true]

4b. Roles and responsibilities among operating partners are well understood.

[Not at all true, somewhat true, mostly true, completely true]

4c. Operating partners are frequently involved in making decisions about the project.

[Not at all true, somewhat true, mostly true, completely true]

4d. Operating partners communicate regularly with each other.

[Not at all true, somewhat true, mostly true, completely true]

4e. Operating partners have an easy means of sharing information and data with each other.

[Not at all true, somewhat true, mostly true, completely true]

4f. This project has brought together a diverse set of operating partners.

[Not at all true, somewhat true, mostly true, completely true]

4g. Operating partners work together to develop long-term strategy.

[Not at all true, somewhat true, mostly true, completely true]

We are now going to ask you three short questions about your relationship with each of the operating partners you listed above:

5a. Formal communications can generally be thought of as scheduled, and often recurring, such as presentations, routine meetings, quarterly board meetings, or newsletters. How frequently do you have formal communication with [Partner X]?

[Daily, weekly, monthly, a few times a year, once a year or fewer]

5b. Informal communications can generally be thought of as casual, often on an as-needed basis, such as an email exchange, a direct phone call, or an impromptu meeting. How frequently do you have informal communication with [Partner X]?

[Daily, weekly, monthly, a few times a year, once a year or fewer]

5c. Which of the following characterize [Partner X’s] contribution to the project? Check all that apply:

Service provision, meaning providing direct services to beneficiaries and making referrals
Systems management, such as fiscal management, record-keeping, IT systems, monitoring and evaluation
Organizational support, such as consulting, training, advertising or outreach, overseeing service members and volunteers
Resources, such as providing or sharing funding sources, sharing facility/office space, contributing other materials
Staff
Other: Please describe ________
Don’t Know

We would also like to know about your peripheral supporters. Peripheral supporters are those individuals, groups, and organizations that are not involved in the day-to-day activities of the project but rather contribute or provide support in more infrequent ways. These supporters can range from non-profit organizations and foundations to private businesses to schools and government agencies, to name a few.

Before responding to the questions below, please list the names of up to 5 Operation AmeriCorps project peripheral supporters whose support is necessary to the project:

[Peripheral Supporters] For the project’s peripheral supporters as a group, to what extent are the following statements true?:

6a. This project has brought together a diverse set of peripheral supporters.

[Not at all true, somewhat true, mostly true, completely true]

6b. Peripheral supporters contribute to the success of this project.

[Not at all true, somewhat true, mostly true, completely true]

6c. Peripheral supporters take part in developing long-term strategy.

[Not at all true, somewhat true, mostly true, completely true]

6d. Peripheral supporters advocate for the project to influential leaders.

[Not at all true, somewhat true, mostly true, completely true]

6e. Peripheral supporters advocate for the project in the communities being served.

[Not at all true, somewhat true, mostly true, completely true]

6f. Peripheral supporters have made formal agreements to support the project.

[Not at all true, somewhat true, mostly true, completely true]

We would now like to ask you some questions about the staffing, infrastructure, and resources for your Operation AmeriCorps project. Several of the following questions will refer to “national service members” and “volunteers.” By “national service members” we mean individuals working on your Operation AmeriCorps project who are serving through AmeriCorps State and National, AmeriCorps VISTA, or AmeriCorps NCCC. By “volunteers” we mean both individuals working on your project through Senior Corps as well as individuals from the general community who have chosen to work on your project and are not being paid to do so.

Please indicate whether the following statements are true or not by selecting “Yes” or “No” for each question.

[Staffing levels] Staff positions that are essential to running the project, not including national service members or volunteers, are fully staffed. [Y/N]

If N…Which key positions remain open? [Free response]

[Staffing levels] National service member and volunteer positions that are essential to running the project are fully staffed. [Y/N]

If N…Which of the following have positions that are not fully staffed? Check all that apply:
- AmeriCorps State and National
- AmeriCorps VISTA
- AmeriCorps NCCC
- Senior Corps
- Other volunteers

[Resources] The project is fully resourced, including funding, space, technology, and materials to carry out the project as planned. [Not at all true, somewhat true, mostly true, completely true]

If Not at all…Which of the following is not fully resourced? Check all that apply:
- Funding for Operating Budget
- Space
- Technology
- Materials
- Other. Please explain:_________________

[Financial strategy] The project has a strategy in place to ensure that the project is fully resourced over the course of the Operation AmeriCorps grant. [Not at all true, somewhat true, mostly true, completely true]

[Board] The project has a formal community advisory group, board, or oversight committee of relevant constituents. [Y/N]

If Y…To what extent are the following statements true about your formal advisory group, board, or oversight committee of relevant constituents: [Question matrix w/ radio buttons; not at all true, somewhat true, mostly true, completely true]

[Board] The project has an informal community advisory group, board, or oversight committee of relevant constituents. [Y/N]

Group membership is representative of the community served.
The group contains broad variety of expertise and experience.
Group members provide direction, support, and accountability.

If N to both 11 and 12…Please describe the project’s strategy for providing community oversight and input into the project. [Free response]

[Sustainability] Our Operation AmeriCorps project has a plan for how it will be sustained beyond the Operation AmeriCorps grant. [Y/N]

If Y…Check all that currently apply to this sustainability plan:
- The plan is documented.
- Necessary operating partners have formally agreed to the plan.
- Necessary peripheral supporters have formally agreed to the plan.
- Adequate resources and financing have been formally committed to make the plan viable.
If N…What are the reasons for not having a sustainability plan? [Free Response]

[Tracking Outcomes] Our Operation AmeriCorps project is collecting the data that it needs to know it is on track to achieve its desired outcomes. [Not at all true, somewhat true, mostly true, completely true]
[Achieving Goals] Our Operation AmeriCorps project is able to demonstrate that it is making progress toward its goals. [Not at all true, somewhat true, mostly true, completely true]

The Privacy Act of 1974 (5 U.S.C § 552a) requires that the following notice be provided: The information requested in the Capacity Survey is collected pursuant to 42 U.S.C 12592 and 12615 of the National and Community Service Act of 1990 as amended, and 42 U.S.C. 4953 of the Domestic Volunteer Service Act of 1973 as amended, and 42 U.S.C. 12639. Purposes and Uses - The information requested is collected for the purposes of assessing the implementation of the Operation AmeriCorps grant initiative. Routine Uses - Routine uses may include disclosure of the information to federal, state, or local agencies pursuant to lawfully authorized requests. In some programs, the information may also be provided to federal, state, and local law enforcement agencies to determine the existence of any prior criminal convictions. The information may also be provided to appropriate federal agencies and Department contractors that have a need to know the information for the purpose of assisting the Department’s efforts to respond to a suspected or confirmed breach of the security or confidentiality or information maintained in this system of records, and the information disclosed is relevant and unnecessary for the assistance. The information will not otherwise be disclosed to entities outside of the Corporation for National and Community Service without prior written permission. Effects of Nondisclosure - The information requested is not mandatory.

Capacity Survey Communications

[Survey Invitation Email]

Reply To: [email protected]

Sender: [email protected]

Subject: Operation AmeriCorps Organizational Capacity Survey: Now Open

Dear [PREFIX] [LASTNAME],

I am writing to ask you to participate in a survey being conducted as part of the Operation AmeriCorps national assessment. The survey is being conducted by the Office of Research and Evaluation at the Corporation for National and Community Service [HYPERLINK TO http://www.nationalservice.gov/] (CNCS), a federal agency that engages more than 5 million Americans in service through its core programs—Senior Corps, AmeriCorps, and the Social Innovation Fund. The purpose of the survey is to assess the organizational capacity of each Operation AmeriCorps project. The results of this survey will provide critical information on the extent to which organizations implementing Operation AmeriCorps projects develop, operate, and sustain programmatic activity.

You were identified as the representative of an organization working on an Operation AmeriCorps project that would be most able to comment on the project’s organizational capacity. Your answers to these questions will not be shared outside of this evaluation, and you will not be identified by your responses. All Operation AmeriCorps projects are required to complete this survey.

The survey will take approximately 30 minutes to complete. To take the survey, please click HERE. [HYPERLINK TO URL]

Please feel free to contact us to assist you in either accessing or completing the web survey. For more information about this survey, or for assistance, please contact Joseph Breems ([email protected]; 202-606-6992) or Adrienne DiTommaso ([email protected]; 202-606-3611).

Thank you in advance for participating in this study!

Sincerely,

XXXXX

[Signatory from CNCS]

In accordance with the Paperwork Reduction Act of 1995, the OMB approval number is XXXX-XXXX and the expiration date is MONTH DAY, 201X.

[First and Second Reminders]

Reply To: [email protected]

Sender: [email protected]

Subject: Operation AmeriCorps Organizational Capacity Survey: Please Complete the Survey

Dear [PREFIX] [LASTNAME],

I am writing to remind you to complete the Operation AmeriCorps Organizational Capacity survey being conducted as part of the Operation AmeriCorps national assessment. The purpose of the survey is to assess the organizational capacity of each Operation AmeriCorps project. The results of this survey will provide critical information on the extent to which organizations implementing Operation AmeriCorps projects develop, operate, and sustain programmatic activity.

The survey will take approximately 30 minutes to complete. To take the survey, please click HERE. [HYPERLINK TO URL]

Thank you in advance for participating in this study!

Sincerely,

XXXXX

[Signatory from CNCS]

In accordance with the Paperwork Reduction Act of 1995, the OMB approval number is XXXX-XXXX and the expiration date is MONTH DAY, 201X.

[Third Reminder]

Reply To: [email protected]

Sender: [email protected]

Subject: Operation AmeriCorps Organizational Capacity Survey: Survey Closing Soon

Dear [PREFIX] [LASTNAME],

The period to complete the Operation AmeriCorps Organizational Capacity Survey is closing soon. Please complete the survey as soon as possible.

All Operation AmeriCorps projects are required to complete this survey.

The survey will take approximately 30 minutes to complete. To take the survey, please click HERE. [HYPERLINK TO URL]

Thank you in advance for participating in this study!

Sincerely,

XXXXX

[Signatory from CNCS]

In accordance with the Paperwork Reduction Act of 1995, the OMB approval number is XXXX-XXXX and the expiration date is MONTH DAY, 201X.

[Fourth and Final Reminder]

Reply To: [email protected]

Sender: [email protected]

Subject: Operation AmeriCorps Organizational Capacity Survey: Survey Closing Today

Dear [PREFIX] [LASTNAME],

The Operation AmeriCorps Organizational Capacity Survey is closing today at 11:59 pm EST. Please complete the survey as soon as possible.

All Operation AmeriCorps projects are required to complete this survey.

The survey will take approximately 30 minutes to complete. To take the survey, please click HERE. [HYPERLINK TO URL]

Thank you in advance for participating in this study!

Sincerely,

XXXXX

[Signatory from CNCS]

In accordance with the Paperwork Reduction Act of 1995, the OMB approval number is XXXX-XXXX and the expiration date is MONTH DAY, 201X.

2015 Fall Grantee Interview Clearance

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: XXXX-YYYY)

Shape2 TITLE OF INFORMATION COLLECTION:

PURPOSE: The purpose of this request is to collect qualitative data via interviews with 10 Operation AmeriCorps grantees to learn more about their experience planning and applying for the Operation AmeriCorps grant. Data from these interviews will be analyzed for the purposes of refining and strengthening future Operation AmeriCorps grant competitions. Data from the interviews will also be used to develop a survey and additional interview protocols for the evaluation of Operation AmeriCorps, which will be conducted internally by CNCS staff from the Office of Research and Evaluation.

DESCRIPTION OF RESPONDENTS: The respondents are non-profit and public sector programs receiving Operation AmeriCorps grants for the 2015-16 program year. Specifically, we will conduct interviews with the lead/applicant organization’s point-of-contact identified in the grant application, or another designated point-of-contact.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[ ] Usability Testing (e.g., Website or Software [ ] Small Discussion Group

[ ] Focus Group [X] Other: Interviews___________

CERTIFICATION:

I certify the following to be true:

The collection is voluntary.
The collection is low-burden for respondents and low-cost for the Federal Government.
The collection is non-controversial and does not raise issues of concern to other federal agencies.
The results are not intended to be disseminated to the public.
Information gathered will not be used for the purpose of substantially informing influential policy decisions.
The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: Diana Epstein

To assist review, please provide answers to the following question:

Personally Identifiable Information:

Is personally identifiable information (PII) collected? [ ] Yes [X] No
If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No
If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [X] No

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Operation AmeriCorps grantees	10	1 hour	10 hours

Totals	10	1 hour	10

FEDERAL COST: The estimated annual cost to the Federal government is $0. The interviews will be conducted by CNCS staff from the Office of Research and Evaluation.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [X] Yes [ ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

The universe of potential respondents are the 10 non-profit or public sector organizations selected as grantees for the 2015-16 Operation AmeriCorps program year. We will interview one individual representing each of the 10 grantees.

Administration of the Instrument

How will you collect the information? (Check all that apply)

[ ] Web-based or other forms of Social Media

[X] Telephone

[ ] In-person

[ ] Mail

[ ] Other, Explain

Will interviewers or facilitators be used? [X] Yes [ ] No

CNCS staff members from the Office of Research and Evaluation will conduct the interviews. The interviews will be audio-recorded and transcribed.

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

Shape3

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time and divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

Confirmation of clearance:

Control No	ICR Ref. No / RCF ID	Agency/Sub	Agency ICR Tracking Number	Title	Current Expiration Date	Request Type	Conclusion Action	Concluded Date
3045-0137	201503-3045-001	CNCS		Feedback on Operation AmeriCorps Processes	05/31/2018	Gen IC	Approved without change	08/21/2015

1 CNCS was granted clearance to conduct these interviews in Fall 2015 per Generic IC with control number 3045-0137 and ICR reference number 201503-3045-001. This generic clearance process is used by federal agencies to receive clearance to ask ‘low-burden’ questions that assess agency service delivery (Presidential Memo on Paperwork Reduction Act- Generic Clearances, issued May 28, 2010). See attachment F for confirmation of clearance and attachment A for the protocol document.

2 Sites for which focus groups are feasible are those sites that are able to assist ORE in locating and organizing a group of respondents to participate. CNCS acknowledges that for some sites, such as schools, may require advance permission or clearance to conduct data collection activities, and in these cases, CNCS will work with the organization to obtain necessary permissions.

3 We anticipate that this will be different for each organization. For some, this would be the legal applicant or sub-applicant. For others, it would be an entity contracted or otherwise assigned to perform the project work; this may not be the same organization as the main implementing organization.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	DiTommaso, Adrienne (Guest)
File Modified	0000-00-00
File Created	2021-01-24