SIF National Assessment OMB Part A 4 16 2015

Longitudinal surveys of SIF grantees (intermediaries) and two comparison groups – (1) organizations that applied for SIF but were not funded, and (2) a national comparison sample of U.S. grantmaking nonprofit organizations
More in-depth qualitative interviews with (1) SIF grantees to develop information on objective evidence of organizational change, and (2) a combination of SIF grantees and other stakeholders (subgrantees, program officers, and others) to develop information briefs on specific topics of interest to CNCS and the field.

The study has both strengths and limitations. Strengths include:

The inclusion of two comparison groups provides the opportunity to address questions about SIF grantees relative to similar, highly committed organizations (the non-funded applicant comparison), and to broader trends in the field of nonprofit grantmaking (the national comparison sample)
The combination of multiple data collection strategies – including quantitative survey methodology with more qualitative interview methods -- makes it possible to obtain more specific objective information about change as well as insights into factors that contribute to reported change.

Major limitations of the proposed study include the small number of SIF intermediaries (27) and non-funded applicant organizations (45), and the fact that much of the change is reported retrospectively and depends on self-report of change. These limitations and the approaches to addressing them are described below.

In total, 27 organizations have received funding through the SIF program during the period 2010-2014, and another 45 organizations were rated as compliant and of good quality but were not selected for funding. Additionally, the experience of the initial 2010 SIF cohort appears (based on anecdotal evidence) to be different in important ways from that of later cohorts, and there has been some variation in the mix of SIF-funded organizations over time.

Collection and analysis of survey data for these organizations can provide evidence about the reported experience of this particular group of organizations; however, to the extent that the mix of organizations or the experience of later cohorts is different, the ability to generalize to a larger population of “possible future SIF intermediaries” is limited. Moreover, as noted in discussion between CNCS and OIRA, the relatively small numbers of cases make analyses of the survey data vulnerable to such factors as the presence of outliers in the group.

Approaches to addressing these limitations include: use of histograms and scatterplots to assess data patterns and potential outliers prior to statistical analyses; use of combined qualitative and quantitative methods to obtain more in-depth understanding of survey findings; acquisition of documentary evidence for reported changes; and careful presentation of findings in reports and briefings so that users understand the limitations and the strengths of the data.

The survey collects data on organizations’ experience in 2009 (prior to initial SIF funding) and 2014. Because of the timing of data collection, these data had to be asked retrospectively. To address this limitation, several approaches are used:
- The questions ask about the level of organizational behavior at each time point, and then, if there are reported differences in level, ask about factors that contributed to change. Pretesting indicates that organizations were able and willing to report both high and low initial and current levels. (That is, they did not necessarily report a pattern of increase in key areas from 2009 to 2014.) This provides a degree of face validity for the responses.
- The survey and follow-up interview with SIF intermediaries asks them to provide objective evidence of change, to support their self-report.
- The 2016 survey round will collect prospective longitudinal data for the 2014 cohort of SIF intermediaries and non-funded applicants, and for the 2010-2012 SIF intermediaries. Although the numbers are small (7 new SIF intermediaries in the 2014 cohort), a comparison between the prospective data for the new cohort and the retrospective report for the earlier cohorts will help provide additional perspective on the reports of change.
- Analyses of SIF intermediary progress reports will provide additional information on grantee progress and challenges, as reported at the time of submission of progress reports to CNCS. Although these data are not discussed in this submission because they are already collected and thus entail no additional respondent burden, analyses of these reports provides additional triangulation of survey and interview findings.

A1. Circumstances That Make the Collection of Information Necessary

The Corporation for National and Community Service (CNCS) has implemented the Social Innovation Fund (SIF), a program that combines public and private resources to grow the impact of innovative, evidence-based solutions to improve the lives of people in low-income communities throughout the U.S.

The Social Innovation Fund (SIF) is a federal program intended to foster innovation to transform lives and communities. A program of the Corporation for National and Community Service (CNCS) launched in 2010 following the signing of the Serve America Act (SAA), it is one of the six Obama Administration “tiered-evidence initiatives” embodying the principles of social innovation. To date, SIF has received an approximately $50-70 million annual appropriation from Congress. As a program, it leverages federal funds through public-private collaborations by granting money to highly successful intermediary grantmakers who in turn find, improve, and grow promising community solutions with evidence of successful outcomes in three core areas: youth development, economic opportunity, and healthy futures. (For more information on the SIF and other federal evidence-based programs, see Haskins and Margolis, 2014.)

The SIF is characterized by the unique interplay of six key elements:

It relies on intermediary grantmaking institutions to implement the program – they take on the role of finding, selecting monitoring, supporting, evaluating and reporting on the nonprofit organizations implementing community-based interventions.
It is a tiered-based evidence program that requires all funded programs/interventions to demonstrate at least preliminary evidence of effectiveness, or funding “what works.”
It requires that all programs or interventions implement a rigorous evaluation that will build on their level of evidence.
It charges intermediaries with scaling evidence-based programs – increasing impact within their community or to communities across the country – and as such, grapples with a field-wide challenge of how best to successfully and efficiently do so.
It leverages public-private partnerships to effect large scale community impact in ways that neither a traditional federal grant investment nor a philanthropic grants investment could achieve on its own. This includes it unique leveraged funding model to support nonprofit programs.
It is committed to improving the effectiveness of nonprofits, funders, and other federal agencies by capturing learning and best practices and promoting approaches that will generate the greatest impact for individuals and communities.

To date, the SIF program has implemented four rounds of grantmaking (in 2010, 2011, 2012, and 2014) and has selected a total of 27 intermediaries for funding.

The exhibit summarizes the number of awards by year:

Award year (cohort)	Number of SIF grantees funded in year
2010	11
2011	5
2012	4
2014	7
Total	27

These intermediaries have in turn selected more than 200 nonprofit organizations through open and competitive processes to implement promising interventions in youth development, economic opportunity, and health.

It is important to assess the SIF initiative, both to allow program evaluation and management by CNCS, and to support the larger goal of learning from the Administration’s evidence-based initiatives. To meet the need for information on the implementation and effectiveness of the SIF, CNCS has contracted with ICF International to support CNCS’s Office of Research and Evaluation to implement a National Assessment of the SIF program. The National Assessment will focus on developing information that provides insights into the SIF’s impact on organizations’ use of evidence-based grantmaking strategies, ability and willingness to build the evidence base for their service models, ability to scale their service models, and use of collaborative approaches to address local community needs. The SIF National Assessment will use a quasi-experimental design, in which SIF intermediaries will be compared with two groups that provide a counterfactual and allow examination of differences associated with SIF: 1) non-funded SIF applicants, and 2) other nonprofit grantmaking organizations that have not applied for a SIF grant. The data collection activities will yield data that support the formative evaluation of SIF grantmaking and capacity building and, through the comparisons and counterfactual analyses, the development of information relating to the impact of SIF on capacity building.

The primary approach to data collection is the survey described in this submission. In addition to the survey, interviews will be conducted. Interviews will be conducted with SIF intermediaries to document the objective evidence they have to support their reports of organizational change. (These interviews are discussed below.) In addition, a small number of interviews will be conducted with survey respondents or other stakeholders to obtain a more in-depth understanding of particular approaches or issues. In no case will these interviews collect standardized or identical data or ask the same kind and level of information from 10 or more persons within any 12 month period. For this reason, these additional in-depth interviews are not included in this submission.

A2. Purposes and Uses of the Collected Information

A.2.1. Research Questions and Logic Model

The SIF National Assessment is designed to address five key questions about changes in organizational capacity (skills, attitudes, behavior) that flow out of the initiative’s theory of change:

1. Does SIF increase organizational use of evidence-based grantmaking strategies?

2. Does it increase organizations’ willingness and ability to build the evidence base for high-impact models?

3. Does it increase organizational knowledge of how to scale proven models?

4. Does it strengthen organizational infrastructure to support the growth of proven models?

5. Does it facilitate collaborative approaches to addressing local community needs?

The exhibit below provides an overview of the logic model that guides the National Assessment:

A.2.2 Survey Plan: Comparison Groups and Survey Rounds

The SIF National Assessment survey uses a quasi-experimental design to assess SIF implementation and provide insights into SIF impact. The survey will collect data on support provided by CNCS to intermediaries, support provided by intermediaries to subgrantees, and increase in capacity among intermediaries and subgrantees. The survey will collect both data supporting quantitative measures of capacity and change, and more open-ended qualitative or narrative responses to obtain additional insight on SIF experience. The quasi-experimental design will permit comparison between SIF intermediaries and the two comparison groups to test the null hypothesis that any observed change among intermediaries is not different from change among the larger world of nonprofit grantmaking organizations. To test this hypothesis, the SIF National Assessment survey will collect data from SIF grantees (intermediaries) and two comparison groups:

1. Non-funded SIF applicants; and

2. Other nonprofit grantmaking organizations that have not applied for a SIF grant.

As Haskins and Margolis (2014) note, the social entrepreneur movement in social program funding has fostered a cultural change toward greater reliance on evidence of program effectiveness, both for the selection of programs to fund and for the evaluation of funded programs. Comparing SIF intermediaries with non-funded SIF applicants will allow an assessment of the degree of difference observed in such areas as reliance on evidence of effectiveness among organizations that have shown a commitment to social innovation funding and evidence. The comparison between SIF intermediaries and a probability sample of nonprofit grantmaking organizations that have not applied for a SIF grant will allow an examination of the extent to which SIF participants’ experiences diverge from the trends in the larger grantmaking world.

The first cohort of SIF intermediaries was funded in 2010, with subsequent cohorts funded in 2011, 2012 and 2014. Survey data will be collected from SIF intermediaries, non-funded SIF applicants, and a comparison group of other nonprofit grantmaking organizations in two rounds. In Round 1 (2015), all SIF intermediaries and non-funded applicants from the 2010-2014 cohorts will be surveyed, together with a probability sample of other nonprofit grantmaking organizations. In Round 2 (2016) the SIF intermediaries (all cohorts) and the non-funded 2014 applicants will be surveyed a second time. Non-funded 2010-2012 applicants and other nonprofit grantmaking organizations will not be included in the 2016 survey.

Slightly different versions of the surveys will be administered to different survey groups and across the two survey rounds. When the survey data are received, a merged file will be created that includes the core items for all groups, together with content specific to each group. The groups to be surveyed in each round of data collection and the content of those surveys are described below. The survey version that will be used for each group in each round is noted to facilitate comparison with the proposed survey instruments.

Group	Subgroup/ Cohort	2015 survey administration (Round 1)		2016 survey administration (Round 2)
Group	Subgroup/ Cohort	Survey version	Period covered	Survey version	Period covered
SIF intermediaries	2010-2012 cohort (N=20)	1	2009-2014	5	2014-2015
SIF intermediaries	2014 cohort (N=7)	2	2009-2014 (pre-SIF) and early SIF months	5	2014-2015
Non-funded applicants	2010-2012 cohort (N=33)	3	2009-2014		No survey
Non-funded applicants	2014 cohort (N=12)	3	2009-2014	6	2014-2015
Nonprofit comparison group (n-400)		4	2009-2014		No survey

Round 1 of the survey (2015) will collect data from (1) all SIF grantees, (2) all non-funded applicants, and (3) a nonprofit comparison group. This survey round will ask respondents to report information for two points in time: 2009 and 2014. Requesting information about 2009 will provide retrospective baseline data reflecting the organizations’ perceived situation prior to award of the first SIF grants. Requesting information about 2014 will provide information about the organizations’ recent situations. This will allow for an analysis of change between 2009 and 2014 and, for areas where change is reported, allow the survey to collect information about respondents’ perceptions of the reasons for the change.

Round 2 (2016) will be administered to all SIF grantees and the 2014 cohort of non-funded applicants to collect information for 2014-2015. Round 2 of the survey will not be administered to two of the groups that will be surveyed in Round 1: (1) the sample of other nonprofit organizations, and (2) the non-selected SIF applicants from the 2010-2012 cohorts. The sample of other nonprofit organizations will have reported their experiences between 2009 and 2014 during Round 1, and this will provide the key information needed to test the study hypotheses related to other nonprofits. For the 2010-2012 non-selected applicant group, key information will have been collected during Round 1. Because it is expected that additional change in 2014-2015 will be relatively modest and incremental for both of these groups, the burden that a second round of data collection would impose does not appear to be warranted by the modest additional information they might be able to contribute to the National Assessment.

In each round of the survey, all respondents will be asked a core set of questions about organizational capacity/behavior and change. These core questions will ask about the use of evidence-based grantmaking strategies; organizations’ implementation and use of rigorous evaluations; organizational infrastructure, especially for evaluation; scaling up of programs based on evidence of effectiveness; knowledge sharing; collaboration to address community needs; and experience with federal funding.

Each survey group will also be asked selected questions targeted to their particular situations in addition to the core questions. These tailored questions will require six separate versions of the survey for various groups, as follows:

Round 1

Survey 1 (SIF Intermediaries, 2010-2012 Cohorts). The SIF intermediaries will be asked the core questions, as well as questions that collect additional information specific to the SIF experience; the additional questions ask about the support they have received to carry out the SIF initiative, the capacity of their subgrantees, and factors that contributed to reported increases in subgrantee capacity, as well as several questions about the SIF initiative and its contributions and challenges.
Survey 2 (SIF Intermediaries, 2014 Cohort). When Round 1 of the survey is administered in 2015, the 2014 SIF grantee cohort will have had only about 6 months of SIF experience, while the 2010-2012 cohorts will have multiple years of SIF experience behind them. For this reason, the 2014 cohort will be asked to answer the core questions with respect to their 2009 to 2014 experience up to the point when they began receiving SIF funding. In a separate section, the 2014 cohort survey also will ask about support received to carry out the SIF initiative during the early months of their SIF experiences and their reflections on SIF based on these months of experience.
Survey 3 (Non-Funded Applicants). The non-funded SIF applicants will be asked the core questions and a small number of additional questions (e.g., areas of activities they fund) and about any effects they believe applying for SIF has had on their organization.
Survey 4 (Nonprofit Comparison Group). The other nonprofit organizations will be asked the core questions and a small number of additional questions (e.g., awareness of SIF, their areas of funding).

Round 2

Survey 5 (SIF Intermediaries, All Cohorts). In round 2, each SIF intermediary will be provided with the information the organization submitted in the Round 1 survey. They will be asked to update the information based on their experiences in the intervening year, and to comment on reasons for any changes. They also will be asked about other SIF-related topics, including support received, development of subgrantee capacity, and observations on the SIF experience.
Survey 6 (Non-Funded Applicants, 2014 Cohort). In round 2, each non-funded applicant from the 2014 cohort will be provided with the information the organization submitted in the Round 1 survey and will be asked to update the information based on their experiences in the intervening year, and to comment on reasons for any changes. They also will be asked about perceived effects of applying for SIF on the organization, from the perspective of additional time.

In addition to providing information to address the key research questions, Round 2 will collect information that will help CNCS and other audiences understand more about the process and timing of change associated with SIF. For example, SIF programmatic experience and comments from grantees in the pilot test indicate that substantial change occurs in the first year or two of SIF funding, although it also is likely that additional change occurs in later years, either as earlier change is further developed or as new areas are addressed (e.g., sustaining the work after the end of SIF funding). The Round 2 information provided by the 2014 cohort, reporting on the first years of funding, and the 2010-2012 cohorts, reporting on the later years of funding, will provide useful insights into the time patterns of change.

Anecdotally, some non-funded SIF applicants have reported that the experience of applying for a SIF grant has brought about change in their organization. Round 1 of the survey asks the 2010-2012 cohorts of non-funded applicants whether the application process resulted in organizational change. The 2014 cohort is asked this question in 2015 and again in 2016 to see if there is more change. This is of interest to CNCS and other funders because it may yield insights into how the implementation of evidence-based funding initiatives, like the SIF, contribute more broadly to change in the field. It is recognized that these findings will be preliminary, both because they are self-reported and because the numbers of organizations is not large; however, with this caveat, it is anticipated that valuable insights will be obtained.

A.2.3 Research Questions and Survey Questions

The research questions and logic model are presented in Section A.2.1, above. The exhibit below lists the research questions to be addressed by the SIF National Assessment survey and the specific survey questions that will obtain the information needed to address each research question. Analysis methods are presented in Section A.16 and in Part B.

In the exhibit, the survey question numbers refer to the location of the question in Version 1 of the survey (the version administered to 2010-2012 SIF grantees in Round 1), unless otherwise noted. The two right hand columns indicate which questions will be asked in each Round, and the Notes section indicates which group(s) will be asked each question. Some survey items provide information related to more than one research question; these are listed under each of the research questions they apply to. A number of questions (e.g., ones relating to grantee selection, support, collaboration and other topics) use a Likert scale to record the extent of organizational practices in these areas in 2009, 2014 and (for the follow-up surveys) 2015. Similarly, for information collected from SIF intermediaries about SIF subgrantee capacity, a Likert scale is used to assess the level of subgrantee capacity at time of SIF entry and the survey. Because this question format is used in most questions, it is not indicated in the table below. Because this question format is used in most questions, it is not indicated in the table below. For questions that collect different information, the particular information collected is noted in parentheses after the question. (For additional information on the questions and their use in the analyses, see A.16.2 Analysis Plan.)

Research Questions and Survey Items	Notes		Rd 1			Rd 2
Does SIF increase organizational use of evidence-based grantmaking strategies?
Selection of grantees to fund
1A. Use of open competitive process	Asked of all respondents.		x			x
1B. Requirement for evidence of effectiveness	Asked of all respondents.		x			x
1C. Requirement for rigorous evaluation plan	Asked of all respondents.		x			x
23. Major changes experienced in successive years of SIF funding (year and reasons)	Asked only of SIF intermediaries.		x			x
26. Effects of SIF on conduct of other grant programs (yes/no, and describe)	Asked only of SIF intermediaries.		x			x
Does it increase organizations’ ability and willingness (skills, attitudes, behaviors) to build the evidence base for high-impact models?
Support provided to grantees funded by organization
2A. Funding for grantee evaluation	Asked of all respondents.			x		x
2B. Technical assistance for grantee evaluation	Asked of all respondents.			x		x
2C. Technical assistance for grantee implementation	Asked of all respondents.			x		x
Organization’s conduct and use of rigorous evaluation
3A. Conduct rigorous evaluation of programs	Asked of all respondents.			x		x
3B. Use evaluation findings to improve programs	Asked of all respondents.			x		x
3C. Use evaluation findings to communicate program effectiveness	Asked of all respondents.			x		x
Support received by SIF intermediaries
9. Support received to increase capacity (source; helpfulness)	Asked only of SIF intermediaries.			x		x
10. Less helpful support; ways to improve	Asked only of SIF intermediaries.			x		x
11. Areas where would have benefited from additional support	Asked only of SIF intermediaries.			x		x
23. Major changes experienced in successive years of SIF funding (year and reasons)	Asked only of SIF intermediaries.			x		x
SIF subgrantee capacity development
12A. Implement interventions	Asked only of SIF intermediaries.		x			x
12B. Conduct rigorous evaluations	Asked only of SIF intermediaries.		x			x
12C Use evaluation findings for program improvement	Asked only of SIF intermediaries.		x			x
12D. Raise matching funds	Asked only of SIF intermediaries.		x			x
12E. Meet compliance requirements	Asked only of SIF intermediaries.		x			x
12F. Scale-up intervention	Asked only of SIF intermediaries.		x			x
12G. Share knowledge	Asked only of SIF intermediaries.		x			x
13. SIF contribution to subgrantee capacity development	Asked only of SIF intermediaries.		x			x
14. Other capacity areas	Asked only of SIF intermediaries.		x			x
SIF intermediary capacity development
15. Factors other than SIF that contributed to capacity change (open-ended)	Asked only of SIF intermediaries.		x				x
16. Challenges to increasing subgrantee capacity, and effective approaches (open-ended)	Asked only of SIF intermediaries.		x				x
23. Major changes experienced in successive years of SIF funding (year and reasons)	Asked only of SIF intermediaries.		x				x
26. Effects of SIF on conduct of other grant programs (yes/no, and describe)	Asked only of SIF intermediaries.		x				x
27. SIF elements organization expects to sustain (open-ended)	Asked only of SIF intermediaries.		x				x
SIF and other tiered evidence programs
20. Contribution to building evidence (amount and description)	Asked only of SIF intermediaries.			x			x
21. Strengths/contributions (open-ended)	Asked only of SIF intermediaries.			x			x
22. Problems/limitations (open-ended)	Asked only of SIF intermediaries.			x			x
Does it increase organizational knowledge of how to scale proven models?
7A. Undertake efforts to scale up existing programs	Asked of all respondents.		x		x
7B. Select programs for scale-up based on rigorous evaluation evidence	Asked of all respondents.		x		x
23. Major changes experienced in successive years of SIF funding (year and reasons)	Asked of all respondents.		x		x
26. Effects of SIF on conduct of other grant programs (yes/no, and describe)	Asked only of SIF intermediaries.		x		x
Does it strengthen organizational infrastructure to support growth of proven models?
4. Staff positions; evaluation partner; have evaluation budget in 2009 and 2014 (yes/no)	Asked of all respondents.		x		x
5. Evaluation budget (total and % of budget) – compare 2014 with 2009	Asked of all respondents.		x		x
6. Evaluation budget – total, %, and staff – in 2009 and 2014	Asked of all respondents.		x		x
23. Major changes experienced in successive years of SIF funding (year and reasons)	Asked only of SIF intermediaries.		x		x
Does it facilitate collaborative approaches to addressing local community needs?
8A. Participate in funding alliances with other nonprofits	Asked of all respondents.		x		x
8B. Participate in collaborations to share knowledge	Asked of all respondents.		x		x
8C. Collaborate to develop public support for programs	Asked of all respondents		x		x
23. Major changes experienced in successive years of SIF funding (year and reasons)	Asked only of SIF intermediaries.		x		x
26. Effects of SIF on conduct of other grant programs (yes/no, and describe)	Asked only of SIF intermediaries.		x		x
Other Information Collected
Organizational demographics
Geographic vs. issue funding focus	Asked of other nonprofits (available for SIF intermediaries and applicants from CNCS sources)		x
Focus of funding (opportunity, youth, community health)	Asked of other nonprofits (available for SIF intermediaries and applicants from CNCS sources)		x
Other information collected from SIF applicants and/or other nonprofits
Awareness of SIF (scale)		Asked only of other nonprofits	x
Effect on organization of participation in SIF application process (open-ended)		Asked only of non-funded SIF applicants	x		x
Additional reflections on SIF program/experience (asked of SIF intermediaries)
24. What are the strengths/benefits of the SIF model compared with other programs you have participated in? Please provide examples. (Open-ended)	Asked only of SIF intermediaries.		x		x
25. What are the challenges/problems of the SIF model compared with other programs you have participated in? Please provide examples. (Open-ended)	Asked only of SIF intermediaries.		x		x
28. What advice would you give a federal agency that was considering use of an intermediary model similar to SIF? (Open-ended)	Asked only of SIF intermediaries.		x		x
29. What recommendations do you have for improving the SIF program? (Open-ended)	Asked only of SIF intermediaries.		x		x

The information collected from the surveys will allow CNCS SIF program administrators and others in the field to understand how the SIF program is implemented and what effects the program has on participating organizations and on the larger field of social innovation, scaling of interventions, and evidence-based grantmaking. The primary user of the findings will be CNCS. In addition, the findings will be useful to other federal leadership, current and potential SIF grantees, the larger nonprofit sector, and the public.

A3. Use of Information Technology and Burden Reduction

Several efforts have been made to minimize the burden on respondents. In designing the survey, each question was carefully considered to avoid duplication of data available from other sources. The burden will be further minimized by making sure that participants only receive questions that are relevant to them; this will be accomplished by administering separate versions of the survey to different respondent groups (as outlined in the exhibit in section A.1) and employing skip patterns. As described above, a determination was made that the second round of the survey would not be administered to the nonprofit sample and the 2010-2012 non-funded SIF applicants; this helps minimize the burden the survey will place on organizations.

The survey will be administered primarily online by emailing participants a link to an online survey tool. Online administration will lessen the burden on participants by allowing them to complete the survey at a time and in a place that is most convenient to them. Additionally, respondents will have the option of completing the entire questionnaire at once, of coming back to it if they are interrupted, and of obtaining assistance from colleagues within the organization to assist with specific sections (e.g., budget information). Using an online survey format also reduces burden by eliminating the time and effort required to mail back a hard-copy survey.

Respondents will be given the option of requesting a hard-copy version of the survey. However, based on the pretest and on the fact that the organizations asked to respond are relatively large organizations, it is estimated that only a very small number of respondents will request hard copies.

A4. Efforts to Identify Duplication

The survey does not duplicate a current data collection activity and there is no information currently available to answer the research questions of the study.

A5. Methods to Minimize Burden on Small Entities

This collection request does not involve burden to small businesses or other small entities.

A6. Consequences of Less Frequent Collection

Without this data collection CNCS will not be able to address the full range of SIF National Assessment issues. The information from this survey is essential for providing insights into the impact of the investment of agency resources in the SIF.

A7. Special Circumstances Influencing Collection

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

A8. Comments in response to the Federal Register Notice and Consultation Outside the Agency

A 60-day public comment Notice was published in the Federal Register on May 2, 2014 (Vol. 79, No. 85, pp. 25115-25118). The comment period ended July 1, 2014. No public comments were received. CNCS invited comments from SIF intermediaries at the time of the 60-day notice; no comments were received in response to that invitation.

SIF senior leadership and Program Officers were consulted for input on the assessment and survey approach and on the issues addressed by the survey. In addition, several persons outside CNCS were consulted to obtain their views on data availability, data elements to be collected, and other issues relating to data collection. Those discussions with several SIF grantees and experts in the field provided insights on user information needs, data availability, and related topics. A pilot test of the survey and administration protocols was conducted with nine organizations. That pilot and revisions based on the pilot are described in Part B (B4. Tests of Procedures or Methods to Be Used).

A9. Explanation of Any Payment or Gift to Respondents

No payments or gifts will be given to respondents for participation in the survey. Organizations that participate in the survey will be offered a copy of the survey report and a benchmarking report that compares their data with aggregate data for other organizations.

A10. Assurance of Confidentiality Provided to Respondents

The introduction to the survey will inform respondents about the purpose of the data collection, the uses to which data will be put, and how respondent identities will be protected. The language includes:

Data will be reported in aggregate; reports of survey findings will not identify individual persons or organizations. If some comments by respondents would be helpful to present with the organization identified, we will check with respondents and only identify the source with the respondent’s permission.

In addition, all survey instruments will include the required Privacy Act notice.

The survey will be administered in an online format using SurveyGizmo Web-based survey software. Survey Gizmo’s survey hosting environment has been designed with security as a foremost consideration, with a 256-bit encryption and secure links in its applications. SurveyGizmo is also fully PCI compliant and scans daily for security threats. Participant email addresses will be uploaded into SurveyGizmo and each respondent will be sent a link to the survey via their email address. All data exported from SurveyGizmo will be kept in a secured folder, and identifying information (names, email address, IP addresses, etc.) will be stripped before reporting.

A11. Justification for Sensitive Questions

There are no questions pertaining to sensitive topics such as sexual behavior and attitudes, religious beliefs, or other matters that are commonly considered private. The survey asks responding organizations about grantee selection, support, evaluation, and similar topics related to the implementation and reported impact of the SIF program.

A12. Estimate of Annualized Hour Burden and Costs to Respondents

Data collection will take place annually for two years. The first year (2015) includes the survey of SIF intermediaries, non-funded applicants, and a probability sample of other nonprofit grantmaking organizations; the second year (2016) only includes the survey of SIF intermediaries and non-funded 2014 applicants.

The average time burden per participant was estimated based on a pilot test of the survey instrument with six respondents to determine the amount of time necessary to complete all questions. Because some of the SIF intermediaries indicated they would need an additional person to help complete the survey, the number of respondents for SIF intermediary surveys was estimated at 1.5. (For this reason, the total number of respondents is somewhat greater than the number of organizations surveyed.) The burden estimates for the SIF intermediaries also include the time for a follow-up interview to review objective evidence for reported changes. These interviews are estimated to take 30 minutes per interview, and will be conducted with one staff member from the intermediary organization (20 interviews in 2015 and 27 interviews in 2016). The number of survey respondents for each year is based on the number of organizations and the number of respondents per intermediary. It is likely that the same person may respond for the organization in each of the two years. Because of this, the total number of separate respondents for the two years is likely to be less than the totals shown in the table; however, the estimates of hours and cost burdens, based on these numbers of respondents, are the ones anticipated for the two survey rounds. The hourly cost in respondent time was calculated based on salary information for recent position listings in Idealist for positions such as Director of Research and Evaluation, and Program Manager; 40 percent was added to the salary figures to account for the costs to the organization of providing benefits, and a 4 percent salary increase factor was added for 2016. The total response burden and costs for this information collection are provided in the exhibit below. The burden and costs shown in the exhibit are the totals for all categories of respondents for each survey year and the annualized burden and costs over the two-year period.

Estimated Respondent Burden
Survey	Number of Respondents	Average Burden Per Respondent	Estimated Total Burden Hours	Hourly Rate	Total Cost
2015 Respondents	486	23.2 minutes	188.2	$63.30	$11,913
2016 Respondents	53	47.7 minutes	42.1	$65.85	$2,772
Totals (2015 + 2016)	539		230.3		$14,685
Annualized	270	25.6 minutes	115.2		$7,343

A13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

The only cost to respondents will be the time it takes to respond to the information collection. This information collection does not involve any other costs to respondents or any recordkeeping.

A14. Estimated Annualized Cost to Federal Government

The estimated cost to the Federal Government for the SIF National Assessment data collection activities is $448,198 over a three year period; the annualized cost to the government is $149,399 over three years. This is the cost of CNCS’s Federal contractor, ICF International, for data collection activities associated with this submission.

A15. Reason for Change in Burden

There are no changes. The survey described in this supporting statement comprises a new information collection.

A16. Plans for Tabulation and Publication of Results and Project Time Schedule

The survey will collect data on the three study groups. For use in analyses, survey data will be linked to other data used in sample selection and data available from CNCS records. Data will be securely stored at CNCS’s contractor, ICF International, in a data system created for this study. CNCS and its contractor will analyze the data and present findings in aggregate form. The analyses will examine changes in organizational capacity and behavior over time, to compare SIF intermediaries with the other two groups and test hypotheses about the effects of the SIF on organizations.

A.16.1 Project Timeline and Publication Plans

Findings will be presented in an annual technical report and in research briefs or other products developed for dissemination to a wider audience of policy makers, grantmaking organizations, and other potential users.

The schedule for information collection, data analysis, reporting, and dissemination of results is shown in the exhibit below:

Activity	Time frame
Round 1 Survey
Data collection	May-June 2015
Data analysis	June-July 2015
Report preparation	July-August 2015
Dissemination	August-September 2015
Round 2 Survey
Data collection	May-June 2016
Data analysis	June-July 2016
Report preparation	July-August 2016
Dissemination	August-September 2016

A.16.2. Analysis Plan

The analysis plan for the SIF survey data is based on the five SIF research questions listed in Section A.2. This section describes the plan for analyses of these data. More detailed information on statistical data analysis is presented in Part B.

Analyses of change in SIF intermediaries’ and other organizations’ capacity. The analytic approach that will be used to address the SIF research questions involves tests of hypotheses regarding change in SIF-participant organizations’ capacity and behaviors compared with those of other organizations surveyed. The analyses will be organized around the core areas of inquiry: the use of evidence-based grantmaking; building the evidence base for promising interventions; scaling evidence-based programs; collaboration and partnerships; and knowledge sharing. The underlying expectation is that SIF intermediaries will experience more positive change and greater change than the other organizations surveyed. It is expected that all organizations are likely to have experienced some change in recent years, as a result of such factors as greater emphasis on evaluation in the grantmaking world, but that the change will be greater for SIF intermediaries. This will be tested using tests of the null hypothesis of no difference between the groups, specifically, that observed change in SIF intermediaries is no greater than would have occurred in the absence of the SIF.

To obtain information needed for the hypothesis tests, the survey will collect information about different aspects of the surveyed organizations’ capacity and behaviors at two points in time, measured with a 7-point Likert scale. Initial data analysis will be completed with Round 1 data. After the Round 2 survey is conducted, the Round 1 and Round 2 data will be merged. This merged dataset will be used for an additional series of data analyses that address issues relating to the timing, magnitude and pace of change experienced.

The analyses will be carried out in several steps. Prior to conducting any significance testing, histograms and scatterplots will be used to visually assess the distributions of the variables to be analyzed. These assessments will provide insight into possible patterns in the data as well as indicate potential outliers. Outliers will be identified on a case-by-case basis, and because they may exert an inordinate effect on group means given the small sample sizes of the intermediary and non-funded applicant groups, these data points may be trimmed or excluded as needed to ensure the accuracy of group statistics. Next, descriptive analyses will be used to compare the SIF intermediaries with the other two groups; these analyses will be carried out using direct comparisons of measures (e.g., means, percentages), and tests of differences between groups.

Difference scores will be calculated after Round 1 by subtracting the 2009 scale value from the 2014 scale value, and after Round 2 by subtracting the 2014 from the 2015 score.

Based on the difference scores, analyses will be conducted to test the differences between the SIF intermediaries and the other two groups. These will use “difference in difference” measures (comparing the difference scores). In each case, the key comparison will be between the SIF intermediaries and one of the other groups. The test of differences between the SIF intermediaries and the non-funded applicants will provides a comparison with the most similar organizations – ones that are highly motivated to undertake innovative, rigorous grantmaking and evaluation strategies to grow promising programs. The test of differences between the SIF intermediaries and the nonprofit sample will addresses the question of whether, how, and to what extent the SIF intermediaries differ from the rest of the nonprofit grantmaking world. This will help answer the question of whether the SIF intermediaries have progressed farther than nonprofits not involved with the SIF in terms of such elements as implementing evidence-based grantmaking strategies, requiring and supporting rigorous evaluation to build the evidence base, and building the infrastructure for supporting innovation and scaling.

A separate comparison between the 2010-2012 SIF intermediary cohorts and the 2014 SIF cohort will allow examination of the change that takes place in the early period of SIF participation compared with the later SIF period – that is, whether the most rapid change in intermediary capacity and behavior occurs in the first year of funding, and what changes continue to occur in later years of funding.

Descriptive analyses of the factors that respondents report contributed to identified areas of change also will be undertaken. Because the factors reported as contributing to change are in the format “mark all that apply,” the analyses will be able to document the contribution of the SIF and other factors to change in SIF intermediary capacity, and provide an in-depth picture of how SIF participation and other factors have contributed to observed change.

Although the extent to which it will be possible to apply multivariate analytic methods will be limited by the number of cases in the SIF intermediary and applicant groups, logistic regression (for dichotomous variables) and ordinal regression (for ordered variables) will be used to control for between-group differences or for the effect of other contextual factors (e.g., economic conditions). These will be exploratory analyses to help develop insights into SIF impacts on organizational change, and will be used and reported with appropriate discussion of data limitations.

Other analyses. Other information collected in the survey includes open-ended or narrative responses to questions that ask about such topics as the support SIF intermediaries have received; the ways SIF and other factors have contributed to the development of intermediary and subgrantee capacity; and major changes SIF intermediaries experienced in successive years of SIF participation. These data will be analyzed thematically, using ATLAS.ti to organize and code the information. The primary purpose of these data and their use in the analyses will be to provide more in-depth understanding of the SIF experience and provide examples of lessons learned or promising practices for achieving change.

A17. Display of OMB Expiration Date

The OMB approval number and expiration date will be displayed on the survey that is completed by participants for this study.

A18. Exceptions to Certification Statement

CNCS does not request an exception to the certification of this information collection.

References

Baruch, Y, and Holton, B.C. (2008) Survey response rates and trends in organizational research. Human Relations 61:1139-1161.

Haskins, Ron and Margolis, Greg (2014) Show Me the Evidence: Obama’s Fight for Rigor and Results in Social Policy. (Washington, D.C.: Brookings Institution Press)

Saunders, M.N.K. (2012) Web versus mail: The influence of survey distribution mode on employees’ response. Field Methods 24:1-18.

Hager, M.A., Wilson, S., Pollak, T.H. and Rooney, P.M. (2003) Response rates for mail surveys of nonprofit organizations: A review and empirical test. Nonprofit and voluntary sector quarterly 32:252-267.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Wilson, Alicia
File Modified	0000-00-00
File Created	2021-01-25

SIF National Assessment OMB Part A 4 16 2015