2 Survey of 2014 SIF Intermediaries

SIF National Assessment Survey
Version 2: 2015 survey of 2014 SIF Intermediaries

Introduction

CNCS’s Office of Research and Evaluation has contracted with ICF International to help conduct a National Assessment of SIF. To understand the SIF program in the larger context of grantmaking in the U.S., ICF is conducting a survey of both SIF intermediaries and a national sample of nonprofit organizations that make grants to other U.S. nonprofits. This survey will be supplemented by more in-depth interviews with SIF intermediaries and others.

Participation in this survey is voluntary, but we hope you will participate because your organization’s SIF experience and perspective are extremely valuable for understanding grantmaking in the U.S., the role of SIF, and ways to improve SIF.

The survey asks about selection of grantees, support for grantees, evaluation, scaling up of programs, and collaboration. For purposes of understanding change in grantmaking over recent years, the survey will ask about your organization both in 2009 and five years later, in 2014. In addition, it asks about support you have received to carry out your work as a SIF intermediary. A similar survey will be sent to your organization in 2016 to ask about the period 2014-2015.

The survey is sent to you as the SIF contact person for your organization. If you need to involve someone else in your organization to respond to the survey questions – either because of the topic or because it asks about 2009 and 2014, please ask that person or persons to respond to questions where they have the needed knowledge. We are requesting that we receive one completed survey for the organization.

The survey will take about 30-40 minutes to complete. Data will be reported in aggregate; reports of survey findings will not identify individual persons or organizations. If some comments by respondents would be helpful to present with the organization identified, we will check with respondents and only identify the source with the respondent’s permission.

If you have any questions about the survey, please contact please contact Elyse Goldenberg ([email protected]; 703-225-2426) or Whitney Marsland ([email protected]; 703-225-2247) at ICF International. 

Selection of Grantees to Fund

This section asks about your organization’s approach to selecting grantees to fund to carry out programs in communities, in both 2009 and 2014. It also asks about changes your organization may have experienced over this period and (when applicable) the major factors that contributed to these changes. For 2014, please answer for the period in 2014 prior to the time when your organization received its SIF funding.

In responding to these questions, please think about your overall funding to nonprofits that conduct programs in communities (not just about your SIF subgrants or about other individual programs or grant portfolios).

1. To what extent did your organization do the following in selecting nonprofits to fund in 2009? To what extent did your organization do these things in 2014 (prior to the time when your organization received its SIF funding)?
   
   For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. For this question, also respond for the period in 2014 prior to the time your organization received its SIF funding. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.
   
   
   

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)

	To what extent did your organization do this in 2014 (prior to the time the organization received SIF funding)?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)

	If applicable: Major factors that contributed to the change between 2009 and 2014 (mark all that apply)
	Trends in the larger grantmaking world	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Used an open competitive process to solicit and review applications and to make selection decisions	[ ]	[ ]	[ ]	[ ]	[ ]
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)	[ ]	[ ]	[ ]	[ ]	[ ]
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to selection of nonprofits to fund between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required prior evidence of intervention effectiveness as basis for funding
C. Required plan for rigorous evaluation of the intervention as a basis for funding

Support for Grantees

Grantmaking organizations vary in the extent to which they provide support to grantees to carry out their work and to develop their capacity to do the work. This section asks about financial assistance your organization may provide to assist your grantees in conducting evaluations of their programs and about non-financial support you may provide to grantees to carry out their work. For 2014, please answer for the period in 2014 prior to the time when your organization received its SIF funding.

The questions use the term “training and technical assistance” to refer to a variety of kinds of non-financial support to help grantees implement their programs and achieve program goals. Examples of training and technical assistance include:

• training and coaching, whether provided in-person or remotely (e.g., webinars)

• technical assistance, including activities such as consultation, problem solving or facilitation

• the provision of handbooks, tools, templates or other resources for grantees to use to carry out their work

• bringing grantees together (in person or remotely) to share problems and solutions

• other similar non-financial support to assist grantees to implement their program

In responding to these questions, please think about your overall funding to nonprofits that conduct programs in communities (not just about your SIF subgrants or about other individual programs or grant portfolios). 

2) To what extent did your organization provide support for your grantees in the following areas in 2009? To what extent did your organization do these things in 2014 (prior to the time when your organization received its SIF funding)? If the support varied by grantee or grant program, please think about your grantees or programs overall or on average.

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. For this question, also respond for the period in 2014 prior to the time your organization received its SIF funding. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program

	To what extent did your organization do this in 2014 (prior to the time the organization received SIF funding)?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program

	If applicable: Major factors that contributed to the change between 2009 and 2014 (mark all that apply)
	Trends in the larger grantmaking world	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)	[ ]	[ ]	[ ]	[ ]	[ ]
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation	[ ]	[ ]	[ ]	[ ]	[ ]
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to support for grantees between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Provided funding to carry out an evaluation or hire an external evaluator
B. Provided training or technical assistance to conduct rigorous evaluation
C. Provided training or technical assistance to support implementation of the program

Evaluation

Evaluation is a systematic process to address such issues as the extent to which a program or intervention achieves its intended outcomes and impacts and how it can be improved. Organizations differ in their use of evaluations of programs that address community needs. In addition, the importance of evaluation as part of an organization's practice may change over time. For 2014, please answer for the period in 2014 prior to the time when your organization received its SIF funding.

This section asks about your organization’s use of evaluation in 2009 and 2014 and its evaluation resources and infrastructure. In addition, we are interested in changes your organization may have experienced over that time, and the factors that contributed to those changes.

3) To what extent did your organization do the following regarding evaluation in 2009? To what extent did you organization do these things in 2014 (prior to the time when your organization received its SIF funding)?

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. For this question, also respond for the period in 2014 prior to the time your organization received its SIF funding. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

	To what extent did your organization do this in 2014 (prior to the time the organization received SIF funding)?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

	If applicable: Major factors that contributed to the change between 2009 and 2014 (mark all that apply)
	Trends in the larger grantmaking world	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Conducted rigorous evaluations of programs funded by your organization	[ ]	[ ]	[ ]	[ ]	[ ]
B. Used evaluation findings to improve programs funded by your organization	[ ]	[ ]	[ ]	[ ]	[ ]
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to evaluation between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

4) Did/does your organization have the following?

	2009	2014 (prior to the time when your organization received its SIF funding)
Staff position(s) or group within your organization dedicated to evaluation	Yes/No	Yes/No
External evaluation partner(s) -- consultant(s) or organization(s) that provide your organization with evaluation services	Yes/No	Yes/No
Part of the organization's budget dedicated to evaluation	Yes/No	Yes/No

5) For this question, think about your organization's total evaluation budget in 2014, prior to receiving your organization’s SIF funding, in comparison to 2009 (both in terms of dollars and as a percentage of the organization's total budget). Was your 2014 evaluation budget…

	Total evaluation budget in dollars	Evaluation budget as percentage of organization’s total budget for year (prior to the time your organization received its SIF funding)
Substantially higher than the evaluation budget in 2009	[ ]	[ ]
Somewhat higher than the evaluation budget in 2009	[ ]	[ ]
About the same as the evaluation budget in 2009	[ ]	[ ]
Somewhat lower than the evaluation budget in 2009	[ ]	[ ]
Substantially lower than the evaluation budget in 2009	[ ]	[ ]

6) If you can access or estimate your organization's budget and staffing for evaluation in 2009 and 2014 (prior to your organization’s receiving its SIF funding), please provide the following information:

	2009	2014 (prior to receiving SIF funding)
Total annual evaluation budget (in $) – for in-house evaluators or external partners
Annual evaluation budget as % of the total organization budget
Number of full time equivalent (FTE) staff with primary responsibility for evaluation

Scaling up of evidence-based programs: Increasing the impact of programs within the community or in other communities

A number of grantmakers are interested in scaling up programs that have shown evidence of effectiveness -- increasing the impact of a program within the community or expanding it to other communities or populations. This section asks about your organization’s involvement in efforts to scale programs. For 2014, please answer for the period in 2014 prior to the time when your organization received its SIF funding.

7) To what extent did your organization do the following regarding scaling up of programs in 2009? To what extent did your organization do these things in 2014 (prior to the time when your organization received its SIF funding)?

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. For this question, also respond for the period in 2014 prior to the time your organization received its SIF funding. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

	To what extent did your organization do this in 2014 (prior to the time the organization received SIF funding)?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

	If applicable: Major factors that contributed to the change between 2009 and 2014 (mark all that apply)
	Trends in the larger grantmaking world	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations	[ ]	[ ]	[ ]	[ ]	[ ]
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization’s approach to scaling up programs between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Undertook efforts to scale up existing program(s)
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

Collaboration to Address Community Needs

8) To what extent did your organization participate in collaborations to support implementation of programs in communities, in 2009? To what extent did you organization do these things in 2014 (prior to the time when your organization received its SIF funding)?

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. For this question, also respond for the period in 2014 prior to the time your organization received its SIF funding. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems

	To what extent did your organization do this in 2014 (prior to the time the organization received SIF funding)?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems

	If applicable: Major factors that contributed to the change between 2009 and 2014 (mark all that apply)
	Trends in the larger grantmaking world	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)	[ ]	[ ]	[ ]	[ ]	[ ]
B. Participated in collaborations with other nonprofit organizations to share knowledge	[ ]	[ ]	[ ]	[ ]	[ ]
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Participated in funding alliance(s) with other nonprofit sector organizations
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy

Support Received by SIF Intermediaries

The preceding sections asked about change in your organization’s experience in different areas in the period between 2009 and 2014 prior to the time when your organization received its SIF funding. Now think about technical assistance or similar support (e.g., coaching, facilitation, tools) your organization may have received since your SIF funding period began in 2014 to help you increase capacity or make changes. CNCS would like feedback from intermediaries to learn from intermediary experience and improve SIF services. 

9) What kinds of support or resources have been especially helpful? Who provided them? (Please give examples)

10) Were there kinds of support or resources that have been less helpful to you? How could these be improved? (Please provide examples or suggestions)

11) Were there areas where you would have benefited from receiving more support or resources than you did? (Please provide examples)

Federal Funding

Federal Funding

12) Did your organization receive any federal government funding in 2009?

( ) Yes

( ) No

13) Was your SIF funding the first federal government funding your organization ever received?

( ) Yes

( ) No

14) Did your organization receive any federal government funding in 2014 before receiving the SIF funding in 2014?

( ) Yes

( ) No

Tiered Evidence Grant Programs.

Tiered evidence programs are programs where the funder awards grants based in part on the quality of past evidence and requires grantees to develop a higher level of evidence for the effectiveness of the programs based on rigorous evaluations.

The federal government funds several tiered evidence initiatives.  In addition to SIF, these include the Department of Education’s Investing in Innovation (i3) program, and the Workforce Innovation Fund (WIF) supported by the Departments of Labor, Education, and Health and Human Services.  

15) Based on your experience to date in SIF, how effective do you think tiered-evidence initiatives are in achieving such outcomes as building evidence in an area?

( ) Very effective

( ) Somewhat effective

( ) Not effective

15a. Please explain:

16) What do you think are the strengths or contributions of tiered-evidence initiatives?

17) What do you think are the problems or limitations of tiered-evidence initiatives?

Reflections on SIF experience

These last questions ask you for some final reflections on your organization’s experience with SIF to date.

18. Based on your experience with SIF to date, what are the strengths/benefits of the SIF model compared with other programs you have participated in? Please provide examples.

19. What are the challenges/problems of the SIF model compared with other programs you have participated in? Please provide examples.

20. Thinking about your organization’s programs (other than SIF) in which you fund grantees to carry out programs in communities: has your SIF experience affected how you conduct those other programs?

( ) Yes

( ) No

( ) N/A – do not have any other programs in which organization funds grantees to carry out programs in communities

(If respondents select “Yes” to question 20) Please describe how your SIF experience has affected the way your organization conducts other grant programs.

21. Are there elements of SIF that you expect your organization will sustain over the longer term, after the completion of the period of SIF funding?

( ) Yes

( ) No

( ) Don’t know

(If respondents select “Yes” to question 21) Please describe the elements that will be sustained and how your organization will sustain them.

SIF elements that will be sustained by organization	How SIF elements will be sustained by organization

22. What advice would you give a federal agency that was considering use of an intermediary model similar to SIF?

23. What recommendations do you have for improving the SIF program?

Thank You!

Thank you for taking our survey.

Your response is very important to the Corporation for National and Community Service