Operation AmeriCorps Justification Part A_2016 1 4

Operation AmeriCorps Evaluation

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSIONS

 

A. Justification

A1.  Need for Information Collection

In the 2015 Operation AmeriCorps Notice of Federal Funding Opportunity (NOFO, (CFDA Number 94.025, http://www.nationalservice.gov/build-your-capacity/grants/funding-opportunities/2015/fy-15-operation-americorps-competition) released by the Corporation for National and Community Service (CNCS) in November, 2014, the stated purpose of this new grant opportunity is to “partner with localities to use national service as the transformative catalyst to address a pressing problem.” To further clarify the priorities of the grant, the NOFO specifies that the “pressing problem” outlined in an application will be identified by a “[t]ribal or local leader (mayor or chief executive),” that the solution proposed will be “place-based, coordinated with CNCS and other local organizations,” that it will be addressed “holistically…in a relatively short period of time (no more than two years)” using AmeriCorps State and National, VISTA, and/or NCCC service members, and that the solution will be “transformational in its expected outcome.”¹ A cohort of 10 applicants were awarded the grant in April, 2015 to be disbursed over the course of two years (all grants for the first cohort will terminate in late summer 2017).

Furthermore, the NOFO specifies that a national assessment of the grant program be conducted simultaneous with this first two-year grant cycle – “In order to learn more about the feasibility, structure, and effectiveness of the coordinated grant making approach of Operation AmeriCorps, CNCS will conduct a national assessment of Operation AmeriCorps projects. The national assessment may include a study of the context, structure, organization, and implementation of Operation AmeriCorps; site-specific analyses of innovation and systems change associated with the program, including partnership arrangements, information technology and data sharing, adoption or improvement of evidence-based practices, development of innovative approaches to management and service delivery, and sustainability; and/or other specific topics based on successful grantee program models…All grantees will be required to cooperate with the national assessment as a condition of grant award.”²

To this end the CNCS Office of Research and Evaluation (ORE) plans to conduct a two-year, mixed methods evaluation of Operation AmeriCorps. Information will be collected from both the current cohort of Operation AmeriCorps grantees and from CNCS staff involved in creating and/or managing the Operation AmeriCorps Grant Program. This study intends to address five research questions derived from the stated purpose of the grant as described in the NOFO.

1. To what extent, and in what ways, are the multiple streams of national service used by Operation AmeriCorps grantees integrated and complementing each other?

2. What evidence is there that community capacity is being developed and sustained with the potential to last beyond the Operation AmeriCorps funding period?

3. What early evidence is there that Operation AmeriCorps outcomes are being achieved?

4. To what extent was Operation AmeriCorps successful as a new way of grant making at CNCS?

5. What lessons from Operation AmeriCorps could be applied to other CNCS grant competitions?

Data and analysis will be grouped in two categories: internal, meaning within CNCS (research questions 4 and 5), and external, which pertains to funded projects and the organizational networks implementing them (research questions 1, 2, and 3).

The evaluation will utilize a comparative case study design, with one case study for each of the funded Operation AmeriCorps grantees. Findings will also be synthesized across grantees. To the extent possible, grantees will be clustered by priority area (Priority 1 or 2, as described in the NOFO and indicated in the project application) and intended outcomes (eg. reduction in community energy use, improve post-high school employment, etc.). Given that Operation AmeriCorps is a new grant program for CNCS, the study will focus primarily on the implementation of the program and the solutions currently funded through it. Some attention (Research Question 3) will be given to analyzing preliminary outcomes at the beneficiary and/or community levels.

Data collection will employ qualitative and quantitative methods to provide a comprehensive assessment of each project’s activities and early outcomes. Qualitative data collection will include baseline interviews with sub-grantees in the fall of 2015 and the fall of 2016³; observation⁴ of and interviews and focus groups with grantee staff (includes legal applicant, sub-applicant, and other project representatives designated by applicant), project partners, national service members, and beneficiaries both on-site and over the phone in the spring of 2016 and 2017. CNCS staff were also interviewed in the fall of 2015 to gather their perspective on the initial development and implementation of the grant opportunity. Quantitative data on grantee capacity will be collected twice a year using an online survey. Performance data and member activity data, as well as information on how the project is structured, will be collected from grantees via the required grantee progress report (GPRs), document requests, and data requests (all such data and documents are considered administrative data).

Research Question 1 stems directly from the NOFO, which states that “[s]uccessful applicants will…[c]oordinate with other AmeriCorps, Senior Companion, Foster Grandparent, RSVP, Social Innovation Fund, and Volunteer Generation Fund programs in their communities.”⁵ The opportunity to request AmeriCorps State and National, AmeriCorps NCCC, and/or AmeriCorps VISTA resources in a single application stands in significant contrast to other AmeriCorps grant programs. Although it is not a requirement of the grant that grantees use more than one stream of national service, use of more than one is nonetheless of strongly indicated interest to CNCS. Findings aimed at addressing this question will be descriptive and comparative (between Operation AmeriCorps grantees) in nature using qualitative data.

Research Question 2 is in response to the NOFO requiring that grantees’ solutions be place-based, involve multiple partners, and have transformational goals. In the literature such characteristics were found to be typical of community change initiatives (Robles, 2010; Kubisch et al., 2010; Kubisch et al., 2011) and collective impact initiatives (Turner et al., 2012; Preskill, Parkhurst, and Splansky Juster, 2014). As such, questions and data collection methods used in this Operation AmeriCorps national assessment are modeled on those outlined in the relevant literature pertaining to the evaluation of such initiatives. These focus on assessing the community infrastructure needed to enact and sustain change by studying the capacity of the organizational network involved in the initiative (Provan and Milward, 2001; Zerounian, Shing, and Hanni, 2011); the nature of the relationships between involved organizations (Hogue, 1993; Robles, 2010); the presence and quality of a “backbone” organization (Turner et al., 2012; Preskill, Parkhurst, and Splansky Juster, 2014); and the involvement of the broader community (Kubisch et al., 2011). Data for both this question and Research Question 1 will use original data collection instruments (survey, interviews, and focus groups) as well as administrative data such as member activity logs, partner organization agreements, and other existing documents.

To ensure the integrity of the organizational capacity survey, CNCS researchers conducted a search for validated or frequently used network and organizational capacity assessment tools. The search concluded that there was no one validated instrument that could be relied upon as a comprehensive assessment of such capacity. However, one of the tools identified in the search, the McKinsey Organizational Capacity Assessment Tool (OCAT), proved a convenient and inclusive tool and ultimately formed the basis for the survey, which was augmented to include validated questions from other sources (referenced above). Major changes to the original OCAT assessment tool included modification for question length, adjusting question orientation to a single survey respondent, clarifying and consolidating response option length, consolidation of overlapping or similar questions into a single question, and including questions to assess the capacity of both the main implementing organization and the organizational network implementing the project.

Interview questions were developed using a matrix of research questions and respondents. For each respondent and research question cell, indicators relevant to the research questions were developed, and from there, specific questions were constructed that would elicit information related to those indicators. Additionally, CNCS researchers consulted the literature on place-based initiatives (referenced above) to guide selection of indicators related to organizational capacity, project implementation and sustainability, and organizational structure of successful place-based initiatives.

Research Question 3 is aimed at assessing the beneficiary and/or community-level outcomes that are being produced by the project, and the extent to which the project’s actual outcomes are meeting its goals as outlined in the application. Given that one of the primary expectations of the NOFO is the production of “transformational change” in communities served by the grantee, it is important to evaluate whether such change is indeed happening. Given that this is the first year of the Operation AmeriCorps program, its current cohort of funded projects should be viewed as pilots. As such, an evaluation that focuses on implementation is the most appropriate study type at this point. Neither readiness for an impact evaluation nor specific level of program maturity were prerequisites for application, therefore it would be unreasonable to expect any grantee to be ready for such an evaluation⁶. This research question will therefore be addressed by looking at outcomes data already being collected by the grantees for the purposes of CNCS performance measurement reporting and additional data collected by grantees for outcomes monitoring and reporting. All findings pertaining to this question will be understood as preliminary “early evidence”.

Research Question 4 was included due to the fact that the Operation AmeriCorps grant program is in its pilot phase. CNCS aims to assess whether this grant-making structure and funding opportunity is effective in achieving the goals as outlined in the NOFO for the purposes of deciding on the future use of this grant-making structure as well as improving other grant programs at the agency. It is also in the interest of CNCS to understand the difficulties of implementation at all levels in order to attain a more comprehensive understanding of the program and a more diverse and useful array of lessons for improvement. Therefore, Research Question 4 will assess CNCS internal processes in implementing Operation AmeriCorps. Data will be collected from both grantees (embedded in data collection instruments for the first three questions) and CNCS staff involved in Operation AmeriCorps pertaining to the perceived ease (or difficulty) of implementation and process changes that have resulted.

Research Question 5 is intended to facilitate learning at CNCS beyond the Operation AmeriCorps grant program. No additional data will be collected for Research Question 5 beyond that which will be collected for questions one through four.

A2.  Indicate how, by whom, and for what purpose the information is to be used.

As described in A1, the purpose of the Operation AmeriCorps national assessment is to collect information about the grant program’s implementation and early outcomes in order to monitor progress towards the goals as outlined in the 2015 Operation AmeriCorps NOFO. The information obtained from the national assessment is also critical to informing decision making around CNCS Strategic Goal 4,⁷ particularly regarding the enhancement of information technology systems, grant-making and financial processes, and agency policies to support more effective and efficient agency operations.

It is anticipated that the users of this data will include CNCS internal stakeholders, the grantees and all partners involved in the funded projects, and organizations, communities, and researchers involved in similar initiatives across the U.S. CNCS will use this valuable data to make decisions about funding and systems improvement. The CNCS Office of Research and Evaluation will make final reports available to internal and external stakeholders, and to the public on nationalservice.gov.

External stakeholders, including the grantees and state service commissions, external researchers, and practitioners may use the reports released to improve the funded initiatives, plan for project sustainability beyond Operation AmeriCorps, and better inform the structuring of similar initiatives. These entities may be supplied with interim reports as is relevant and necessary. The CNCS Office of Research and Evaluation will strive to ensure appropriate use of results through explaining the strengths and limitations of the data and the resulting analyses.

The final evaluation report will be posted on nationalservice.gov.

 

 A3.  Minimize Burden: Use of Improved Technology to Reduce Burden

CNCS will assess the Operation AmeriCorps grant program and grantee initiatives through the data collection methodologies outlined above over the course of the two-year grant. The majority of data collection will be via interviews. In order to reduce time and resource burden for the grantees and other relevant parties, these interviews will be conducted largely by phone. The survey referenced previously will be administered via an online platform, further reducing burden on respondents.

A4.  Non-Duplication

The Operation AmeriCorps national assessment will use administrative data and existing documents to address portions of Research Questions 1-3. Data collection has been streamlined into a reduced number of data collection instruments as opposed to developing a separate instrument for each research question. There are otherwise no other sources of information by which CNCS can meet the purposes described in A2 (above).

 

A5.  Minimizing for economic burden for small businesses or other small entities.

This collection of information does not impact small businesses because they are not directly involved in the implementation of Operation AmeriCorps initiatives.

A6.  Consequences of the collection if not conducted, conducted less frequently, as well as any technical or legal obstacles to reducing burden.

 

If unable to collect data through the means outlined above, CNCS will not be able to assess these aspects of Operation AmeriCorps grantees, and their use of federal funds in meeting the expectations of the 2015 Operation AmeriCorps NOFO. CNCS will also not be meeting the intended activities of the grant as stated in the NOFO, namely the administration of a national assessment. Additionally, internal and external stakeholders will lack critical feedback necessary to learn about program implementation and early outcomes, and will be unable to make data-driven decisions about elements of both the grant program and funded initiatives that could be crucial to success at all levels.

A7.  Special circumstances that would cause information collection to be collected in a manner requiring respondents to report more often than quarterly; report in fewer than 30 days after receipt of the request; submit more than an original and two copies; retain records for more than three years; and other ways specified in the Instructions focused on statistical methods, confidentially, and proprietary trade secrets.

There are no special circumstances that would require the collection of information in these ways.

A8.  Provide copy and identify the date and page number of publication in the Federal Register of the Agency’s notice. Summarize comments received and actions taken in response to comments. Specifically address comments received on cost and hour burden.

The 60 day Notice soliciting comments was published on Monday, October 26, 2015 on page 65219-65220. No (0) comments were received.

 

A9.  Payment to Respondents

There are no payments or gifts to respondents.

  

A10.  Assurance of Confidentiality and its basis in statute, regulation, or agency policy.

Individual responses to this information collection will remain private to the extent permitted by law.

The basis for the assurance of privacy is from the privacy statement in the instruments’ instructions (see attached instruments). All analyses, summaries, or briefings will be presented at the aggregate level and it will not be possible to identify individual respondents in any material that is presented.

The survey data will be stored on CNCS’s secure server, which is protected by a firewall that monitors and evaluates all attempted connections from the Internet. Access to any data with identifying information will be limited only to CNCS staff directly working on the survey.

A11.  Sensitive Questions

 

The information collection does not include questions of a sensitive nature.

 

A12. Hour burden of the collection

We expect up to 170 respondents per year to respond to one or more data collection instrument. The frequency of response requests will be no greater than quarterly. Estimated time for response ranges from 0.5 to 1.5 hours, depending on the instrument. The estimated total burden hours equals 210 hours per year, or 420 hours over the course of the full two-year national assessment.

 

Exhibit 1. Operation AmeriCorps National Evaluation - Projected burden hours
	Instrument	Appx. Time to Administrate (Hours)	# of Respondents	# of Administrations per Respondent	Total Burden (Hours)
YEAR 1	Fall Interview	1	10	1	10
	Survey	0.5	10	2	10
	Spring Interview	1.5	80	1	120
	Spring Focus Group	1	70	1	70
	Year 1 Total		170		210
YEAR 2	Fall Interview	1	10	1	10
	Survey	0.5	10	2	10
	Spring Interview	1.5	80	1	120
	Spring Focus Group	1	70	1	70
	Year 2 Total		170		210
	STUDY TOTAL				420

A13. Cost burden to the respondent

There is no cost to the respondent beyond the time needed to complete the survey and participate in interviews or focus groups.

 

A14. Cost to Government

This national assessment is being conducted by existing staff of the CNCS Office of Research and Evaluation using the office’s budgeted programming and staffing funds and as such present no additional cost to the Federal Government.

A15. Reasons for program changes or adjustments in burden or cost.

The second year of the national assessment, in terms of data collection instruments and response request frequency, is currently planned to mirror the first year in an effort to assess any changes over time. This may be adjusted depending on whether all current grantees remain in the program for the full two years.

 

A16.  Publication of results

The data from the Operation AmeriCorps national assessment will be collected, analyzed, and reported both to CNCS and shared publicly.

The data will be used primarily to learn about the implementation and early outcomes of the Operation AmeriCorps grant at the agency, grantee, and beneficiary/community levels. Reporting and dissemination of results will be in the form of written memos and reports, with oral briefings to key stakeholders as requested. Reports and briefings will be focused on information relevant to program improvement and enhancement and the performance of the funded initiatives, with an emphasis on the strengths and limitations of the data and corresponding analyses to ensure appropriate use of results. The data gathered in this assessment may be utilized in analysis and planning work for other program evaluations and research projects conducted by CNCS. 

At minimum, a final report resulting from the Operation AmeriCorps national assessment will be made available to the general public online following the close of the two-year grant cycle.

 

A17.  Explain the reason for seeking approval to not display the expiration date for OMB approval of the information collection.

Not applicable.

 

A18.  Exceptions to the certification statement

There are no exceptions to the certification statement.

References

Hogue, T. (1993). Community-based collaboration: Community wellness multiplied. Oregon Center for Community Leadership, Oregon State University.

Kubisch, A.C., Auspos, P., Brown, P. and Dewar, T. (2010). Community Change Initiatives from 1990-2010: Accomplishments and Implications for Future Work. Community Investments, 22(1). 8-12.

Kubisch, A.C., Auspos, P., Brown, P., Buck, E. and Dewar, T. (2011). Voices From the Field III: Lessons and Challenges for Foundations Based on Two Decades of Community-Change Efforts. The Foundation Review, 3(1). 138-149.

McKinsey Organizational Capacity Assessment Tool. Accessed July, 2015. http://mckinseyonsociety.com/ocat/.

Preskill, H., Parkhurst, M. and Splansky Juster, J. (2014). Guide to Evaluating Collective Impact, Part 1: Learning and Evaluation in the Collective Impact Context. Collective Impact Forum, FSG.

Preskill, H., Parkhurst, M. and Splansky Juster, J. (2014). Guide to Evaluating Collective Impact, Part 2: Assessing Progress and Impact. Collective Impact Forum, FSG.

Preskill, H., Parkhurst, M. and Splansky Juster, J. (2014). Guide to Evaluating Collective Impact, Part 3: Supplement - Sample Questions, Outcomes, and Indicators. Collective Impact Forum, FSG.

Provan, K.G. and Milward, H. B. (2001). Do Networks Really Work? A Framework for Evaluating Public-Sector Organizational Networks. Public Administration Review, 61(4). 414-423.

Robles, A. (2010). Final Report for the Fairfax County Neighborhood Initiatives Program 2007-2009: A Model for Collaborative Community Change, Volume 1. Center for Social Science Research, George Mason University: Fairfax, VA.

Turner, S., Merchant, K., Kania, K. and Martin, E. (2012). Understanding the Value of Backbone Organizations in Collective Impact. Standford Social Innovation Review. FSG.

Zerounian, P., Shing, J. and Hanni, K.D. (2011). Assessing Nonprofit Networks Prior to Funding: Tools for Foundations to Determine Life Cycle Phase and Function. The Foundation Review, 3(1). 43-58.

1 2015 Operation AmeriCorps Notice of Federal Funding Opportunity, p1 < http://www.nationalservice.gov/build-your-capacity/grants/funding-opportunities/2015/fy-15-operation-americorps-competition>.

2 Ibid., pp4-5.

3 CNCS was granted clearance to conduct these interviews in Fall 2015 per Generic IC with control number 3045-0137 and ICR reference number 201503-3045-001. This generic clearance process is used by federal agencies to receive clearance to ask ‘low-burden’ questions that assess agency service delivery (Presidential Memo on Paperwork Reduction Act- Generic Clearances, issued May 28, 2010).

4 Direct observation of fewer than nine grantees will be conducted by ORE during on-site visits.

5 Ibid., p2.

6 Ibid., p4.

7 Goal 4 aims to “Fortify management operations and sustain a capable, responsive and accountable organization”. 2011-2015 Strategic Plan, p23.

10