1 Survey of 2010-2012 Intermediaries

SIF National Assessment Survey
Version 1: 2015 survey of 2010-2012 SIF Intermediaries

Introduction

CNCS’s Office of Research and Evaluation has contracted with ICF International to help conduct a National Assessment of SIF. To understand the SIF program in the larger context of grantmaking in the U.S., ICF is conducting a survey of both SIF intermediaries and a national sample of nonprofit organizations that make grants to other U.S. nonprofits. This survey will be supplemented by more in-depth interviews with SIF intermediaries and others.

Participation in this survey is voluntary, but we hope you will participate because your organization’s SIF experience and perspective are extremely valuable for understanding grantmaking in the U.S., the role of SIF, and ways to improve SIF.

The survey asks about selection of grantees, support for grantees, evaluation, scaling up of programs, and collaboration. For purposes of understanding change in grantmaking over recent years, the survey will ask about your organization both in 2009 and five years later, in 2014. In addition, it asks about change your organization may have experienced as a result of SIF, support you have received, and your support to subgrantees to build their capacity. A similar survey will be sent to your organization in 2016 to ask about the period 2014-2016. We also will plan to contact you shortly to schedule a short telephone call to ask about documentation of changes related to your organization’s SIF participation.

The survey is sent to you as the SIF contact person for your organization. If you need to involve someone else in your organization to respond to the survey questions – either because of the topic or because it asks about 2009 and 2014, please ask that person or persons to respond to questions where they have the needed knowledge. We are requesting that we receive one completed survey for the organization.

The survey will take about 30-40 minutes to complete. Data will be reported in aggregate; reports of survey findings will not identify individual persons or organizations. If some comments by respondents would be helpful to present with the organization identified, we will check with respondents and only identify the source with the respondent’s permission.

If you have any questions about the survey, please contact  Elyse Goldenberg ([email protected]; 703-225-2426) or Whitney Marsland ([email protected]; 703-225-2247) at ICF International. 

Selection of Grantees to Fund

This section asks about your organization’s approach to selecting grantees to fund to carry out programs in communities, in both 2009 and 2014. It also asks about changes your organization may have experienced over this period and (when applicable) the major factors that contributed to these changes.

In responding to these questions, please think about your overall funding to nonprofits that conduct programs in communities (not just about your SIF subgrants or about other individual programs or grant portfolios).

1. To what extent did your organization do the following in selecting nonprofits to fund in 2009? To what extent did your organization do these things in 2014?
   
   For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)

	To what extent did your organization do this in 2014?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)

	If applicable: Major factors that contributed to the change (mark all that apply)
	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Used an open competitive process to solicit and review applications and to make selection decisions	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to selection of nonprofits to fund between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required prior evidence of intervention effectiveness as basis for funding
C. Required plan for rigorous evaluation of the intervention as a basis for funding

1a. If your organization’s participation in SIF contributed to changes in your organization's approach to selecting subgrantees: How did SIF participation contribute to the changes?

Support for Grantees

Grantmaking organizations vary in the extent to which they provide support to grantees to carry out their work and to develop their capacity to do the work. This section asks about financial assistance your organization may provide to assist your grantees in conducting evaluations of their programs and about non-financial support you may provide to grantees to carry out their work.

The questions use the term “training and technical assistance” to refer to a variety of kinds of non-financial support to help grantees implement their programs and achieve program goals. Examples of training and technical assistance include:

• training and coaching, whether provided in-person or remotely (e.g., webinars)

• technical assistance, including activities such as consultation, problem solving or facilitation

• the provision of handbooks, tools, templates or other resources for grantees to use to carry out their work

• bringing grantees together (in person or remotely) to share problems and solutions

• other similar non-financial support to assist grantees to implement their program

In responding to these questions, please think about your overall funding to nonprofits that conduct programs in communities (not just about your SIF subgrants or about other individual programs or grant portfolios). 

To what extent did your organization provide support for your grantees in the following areas, in 2009 and 2014?  If the support varied by grantee or grant program, please think about your grantees or programs overall or on average. For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program

	To what extent did your organization do this in 2014?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program

	If applicable: Major factors that contributed to the change (mark all that apply)
	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to support for grantees between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Provided funding to carry out an evaluation or hire an external evaluator
B. Provided training or technical assistance to conduct rigorous evaluation
C. Provided training or technical assistance to support implementation of the program

2a. If your organization’s participation in SIF contributed to changes in the support your organization provides to grantees: How did SIF participation contribute to the change?

Evaluation

Evaluation is a systematic process to address such issues as the extent to which a program or intervention achieves its intended outcomes and impacts and how it can be improved. Organizations differ in their use of evaluations of programs that address community needs. In addition, the importance of evaluation as part of an organization's practice may change over time.

This section asks about your organization’s use of evaluation in 2009 and 2014 and its evaluation resources and infrastructure. In addition, we are interested in changes your organization may have experienced over that time, and the factors that contributed to those changes.

3) To what extent did your organization do the following regarding evaluation in 2009 and 2014?

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

	To what extent did your organization do this in 2014?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

	If applicable: Major factors that contributed to the change (mark all that apply)
	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Conducted rigorous evaluations of programs funded by your organization	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Used evaluation findings to improve programs funded by your organization	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to evaluation between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

3a. If your organization’s participation in SIF contributed to change in your organization's approach to evaluation: How did SIF participation contribute to the change?

4) Did/does your organization have the following?

	2009	2014
Staff position(s) or group within your organization dedicated to evaluation	Yes/No	Yes/No
External evaluation partner(s) -- consultant(s) or organization(s) that provide your organization with evaluation services	Yes/No	Yes/No
Part of the organization's budget dedicated to evaluation	Yes/No	Yes/No

5) For this question, think about your organization's total evaluation budget in 2014 in comparison to 2009 (both in terms of dollars and as a percentage of the organization's total budget). Was your 2014 evaluation budget…

	Total evaluation budget in dollars	Evaluation budget as percentage of organization’s total budget for year
Substantially higher than the evaluation budget in 2009	[ ]	[ ]
Somewhat higher than the evaluation budget in 2009	[ ]	[ ]
About the same as the evaluation budget in 2009	[ ]	[ ]
Somewhat lower than the evaluation budget in 2009	[ ]	[ ]
Substantially lower than the evaluation budget in 2009	[ ]	[ ]

6) If you can access or estimate your organization's budget and staffing for evaluation in 2009 and 2014, please provide the following information:

	2009	2014
Total annual evaluation budget (in $) – for in-house evaluators or external partners
Annual evaluation budget as % of the total organization budget
Number of full time equivalent (FTE) staff with primary responsibility for evaluation

Scaling up of evidence-based programs: Increasing the impact of programs within the community or in other communities

A number of grantmakers are interested in scaling up programs that have shown evidence of effectiveness -- increasing the impact of a program within the community or expanding it to other communities or populations. This section asks about your organization’s involvement in efforts to scale programs.

7) To what extent did your organization do the following regarding scaling up of programs in 2009 and 2014?

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

	To what extent did your organization do this in 2014?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

	If applicable: Major factors that contributed to the change (mark all that apply)
	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization’s approach to scaling up programs between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Undertook efforts to scale up existing program(s)
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

7a. If your organization’s participation in SIF contributed to change in your organization's approach to scaling up programs: How did SIF participation contribute to the change?

Collaboration to Address Community Needs

8) To what extent did your organization participate in collaborations to support implementation of programs in communities, in 2009 and in 2014?

For each thing that changed between 2009 and 2014, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2009 and 2014, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

	To what extent did your organization do this in 2009?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems

	To what extent did your organization do this in 2014?
	Always	To a very large extent	To a large extent	To a moderate extent	To a small extent	To a very small extent	Not at all
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems

	If applicable: Major factors that contributed to the change (mark all that apply)
	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Participated in collaborations with other nonprofit organizations to share knowledge	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization between 2009 and 2014, specify those factors below:

	Other major factor(s) that contributed to change between 2009 and 2014
A. Participated in funding alliance(s) with other nonprofit sector organizations
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy

8a. If your organization’s participation in SIF contributed to change in your organization's collaboration: How did SIF participation contribute to the change?

Support Received by SIF Intermediaries

The preceding sections asked about change in your organization’s experience in different areas in the period between 2009 and 2014. Now think about technical assistance or similar support (e.g., coaching, facilitation, tools) your organization may have received during your SIF funding period to help you increase capacity or make changes. CNCS would like feedback from intermediaries to learn from intermediary experience and improve SIF services. 

9) What kinds of support or resources have been especially helpful? Who provided them? (Please give examples)

10) Were there kinds of support or resources that have been less helpful to you? How could these be improved? (Please provide examples or suggestions)

11) Were there areas where you would have benefited from receiving more support or resources than you did? (Please provide examples)

Development of Capacity among Your Organization’s SIF Subgrantees

This section asks about your SIF subgrantees’ capacity to carry out different functions - at the time their SIF funding started and in 2014- and the extent to which any change in their capacity were attributable to their participation in SIF.

In responding to these questions, please think about your SIF subgrantees overall or on average. If some subgrantees have experienced particularly great increases in capacity, or have faced particular challenges or have not increased capacity, you can mention those exceptions in examples.

12) Overall, how would you rate your SIF subgrantees' capacity to do the following, at the time their SIF funding started and in 2014? For areas where your SIF grantees’ capacity has increased, how much of that change is the result of their participation in SIF?

	Your SIF Subgrantees’ Capacity at the Time their SIF Funding Started
	Very Strong	Strong	Moderate	Weak	Very Weak
A. Implement the interventions they are carrying out in their communities
B. Design and conduct rigorous evaluations of their interventions
C. Make use of evaluation findings for program improvement
D. Raise matching funds for the intervention
E. Meet federal compliance requirements
F. Scale up the intervention (i.e., increase impact within community, or expand to other communities)
G. Share knowledge and best practices

	Your SIF Subgrantees’ Capacity in 2014
	Very Strong	Strong	Moderate	Weak	Very Weak
A. Implement the interventions they are carrying out in their communities
B. Design and conduct rigorous evaluations of their interventions
C. Make use of evaluation findings for program improvement
D. Raise matching funds for the intervention
E. Meet federal compliance requirements
F. Scale up the intervention (i.e., increase impact within community, or expand to other communities)
G. Share knowledge and best practices

	How much of the change in your subgrantees’ capacity has been a result of their participation in SIF?
	A substantial amount	Some	A little	None	Not applicable (no change)
A. Implement the interventions they are carrying out in their communities
B. Design and conduct rigorous evaluations of their interventions
C. Make use of evaluation findings for program improvement
D. Raise matching funds for the intervention
E. Meet federal compliance requirements
F. Scale up the intervention (i.e., increase impact within community, or expand to other communities)
G. Share knowledge and best practices

13) If there has been a change in your SIF subgrantees' capacity, please provide examples and describe how SIF has contributed to the changes. (These can include reduction in capacity as well as increase.)

14) If there are other areas where SIF subgrantee capacity has increased because of SIF, please describe:

15) If there has been a change in your SIF subgrantees' capacity, what other factors (other than SIF) have contributed to the change? (These can include reduction in capacity as well as increase.)

16) What have been some of the greatest challenges you have experienced in seeking to increase SIF subgrantee capacity, and what approaches have been most effective in addressing these challenges?

	Challenges in seeking to increase SIF sub-grantee capacity	Approaches that have been most effective in addressing challenges
1.
2.
3.

Federal Funding

Federal Funding

17) Did your organization receive any federal government funding (other than SIF) in 2009?

( ) Yes

( ) No

18) Was your SIF funding the first federal government funding your organization ever received?

( ) Yes

( ) No

19) Did your organization receive any federal government funding in 2014?

( ) Yes

( ) No

Tiered Evidence Grant Programs.

Tiered evidence programs are programs where the funder awards grants based in part on the quality of past evidence and requires grantees to develop a higher level of evidence for the effectiveness of the programs based on rigorous evaluations.

The federal government funds several tiered evidence initiatives.  In addition to SIF, these include the Department of Education’s Investing in Innovation (i3) program, and the Workforce Innovation Fund (WIF) supported by the Departments of Labor, Education, and Health and Human Services.  
 

20) Based on your experience in SIF, how effective do you think tiered-evidence initiatives are in achieving such outcomes as building evidence in an area?

( ) Very effective

( ) Somewhat effective

( ) Not effective

20a. Please explain:

21) What do you think are the strengths or contributions of tiered-evidence initiatives?

22) What do you think are the problems or limitations of tiered-evidence initiatives?

Reflections on SIF experience

These last questions ask you for some final reflections on your organization’s experience with SIF.

23. Thinking about the successive years of your SIF funding – what were the major changes your organization experienced in each year of SIF funding and what were the reasons for those changes? (If you have not yet experienced the later years, please write N/A for “not applicable – have not yet experienced that year”)

	Changes Organization Experienced in Year	Major Reasons for Changes
First Year
Second Year
Third Year
Fourth Year
Fifth Year

24. What are the strengths/benefits of the SIF model compared with other programs you have participated in? Please provide examples.

25. What are the challenges/problems of the SIF model compared with other programs you have participated in? Please provide examples.

26. Thinking about your organization’s programs (other than SIF) in which you fund grantees to carry out programs in communities: has your SIF experience affected how you conduct those other programs?

( ) Yes

( ) No

( ) N/A – do not have any other programs in which organization funds grantees to carry out programs in communities

(If respondents select “Yes” to question 26) Please describe how your SIF experience has affected the way your organization conducts other grant programs.

27. Are there elements of SIF that your organization has sustained or will sustain over the longer term, after the completion of the period of SIF funding?

( ) Yes

( ) No

( ) Don’t know

(If respondents select “Yes” to question 27) Please describe the elements that will be sustained and how your organization will sustain them).

SIF elements that will be sustained by organization	How SIF elements will be sustained by organization

28. What advice would you give a federal agency that was considering use of an intermediary model similar to SIF?

29. What recommendations do you have for improving the SIF program?

Thank You!

Thank you for taking our survey.

Your response is very important to the Corporation for National and Community Service.

We will plan to contact you shortly to schedule a short telephone call to ask about documentation of changes related to your organization’s SIF participation. We are interested in documented evidence of change in such areas as organizational practice, policies and procedures, organizational structure, or changes in engagement with other organizations and the field. We will send you an email in advance and schedule a call for a time that is convenient for you.