Form 5 2016 Survey of SIF Intermediaries

SIF National Assessment Survey
Version 5: 2016 survey of SIF Intermediaries

Introduction

CNCS’s Office of Research and Evaluation has contracted with ICF International to help conduct a National Assessment of SIF. In 2015 we sent you a survey asking about your organization’s SIF experience and its work in such areas as grantee selection and support, evaluation, scaling up of programs, and collaboration. It also asked about support your organization received for SIF, change the organization experienced as a result of SIF, and support to subgrantees.

This survey asks you about your organization’s experience in these areas in the period between 2014 and 2015. To facilitate your responses, we have provided the responses for your organization from the earlier survey. What we are asking you to do now is provide information on the same topics for 2015 and change between 2014 and 2015. We also will plan to contact you shortly to schedule a short telephone call to ask about documentation of changes related to your organization’s SIF participation.


Participation in this survey is voluntary, but we hope you will participate because your organization’s SIF experience and perspective are extremely valuable for understanding grantmaking in the U.S., the role of SIF, and ways to improve SIF.

The survey is sent to you as the SIF contact person for your organization. If you need to involve someone else in your organization to respond to the survey questions please ask that person or persons to respond to questions where they have the needed knowledge. We are requesting that we receive one completed survey for the organization.

The survey will take about 30-40 minutes to complete. Data will be reported in aggregate; reports of survey findings will not identify individual persons or organizations. If some comments by respondents would be helpful to present with the organization identified, we will check with respondents and only identify the source with the respondent’s permission.

If you have any questions about the survey, please contact  Elyse Goldenberg ([email protected]; 703-225-2426) or Whitney Marsland ([email protected]; 703-225-2247) at ICF International. 

Selection of Grantees to Fund

This section asks about your organization’s approach to selecting grantees to fund to carry out programs in communities in 2015. It also asks about changes your organization may have experienced over the 2014-2015 period and (when applicable) the major factors that contributed to these changes.

In responding to these questions, please think about your overall funding to nonprofits that conduct programs in communities (not just about your SIF subgrants or about other individual programs or grant portfolios).

1. To what extent did your organization do the following in selecting nonprofits to fund in 2015?
   
   For each thing that changed between 2014 and 2015, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2014 and 2015, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

CUSTOMIZED SURVEY WILL SHOW THE ORGANIZATION’S RESPONSES FOR 2009 AND 2014. THEN IT WILL ASK ABOUT 2015.

THE RESPONSES FOR YOUR ORGANIZATION FOR 2009 AND 2014 WERE:

	Extent to which organization did this in			Major factors that contributed to change between 2014 and 2015 (mark all that apply)
	2009	2014	2015	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Used an open competitive process to solicit and review applications and to make selection decisions	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Required applicant organizations to provide evidence of intervention effectiveness to be eligible for funding (includes pre- and post-test outcome data or other evidence based on evaluation studies)	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible for funding (that is, quasi-experimental designs with a comparison group, experimental designs or other similarly rigorous designs)	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to selection of nonprofits to fund between 2014 and 2015, specify those factors below:

	Other major factor(s) that contributed to change between 2014 and 2015
A. Used an open competitive process to solicit and review applications and to make selection decisions
B. Required prior evidence of intervention effectiveness as basis for funding
C. Required plan for rigorous evaluation of the intervention as a basis for funding

1a. If your organization’s participation in SIF contributed to changes between 2014 and 2015 in your organization's approach to selecting subgrantees: How did SIF participation contribute to the changes?

Support for Grantees

Grantmaking organizations vary in the extent to which they provide support to grantees to carry out their work and to develop their capacity to do the work. This section asks about financial assistance your organization may provide to assist your grantees in conducting evaluations of their programs and about non-financial support you may provide to grantees to carry out their work.

The questions use the term “training and technical assistance” to refer to a variety of kinds of non-financial support to help grantees implement their programs and achieve program goals. Examples of training and technical assistance include:

• training and coaching, whether provided in-person or remotely (e.g., webinars)

• technical assistance, including activities such as consultation, problem solving or facilitation

• the provision of handbooks, tools, templates or other resources for grantees to use to carry out their work

• bringing grantees together (in person or remotely) to share problems and solutions

• other similar non-financial support to assist grantees to implement their program

In responding to these questions, please think about your overall funding to nonprofits that conduct programs in communities (not just about your SIF subgrants or about other individual programs or grant portfolios). 

2. To what extent did your organization provide support for your grantees in the following areas in 2015?  If the support varied by grantee or grant program, please think about your grantees or programs overall or on average.
   
   For each thing that changed between 2014 and 2015, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2014 and 2015, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

CUSTOMIZED SURVEY WILL SHOW THE ORGANIZATION’S RESPONSES FOR 2009 AND 2014. THEN IT WILL ASK ABOUT 2015.

THE RESPONSES FOR YOUR ORGANIZATION FOR 2009 AND 2014 WERE:

	Extent to which organization did this in			Major factors that contributed to change between 2014 and 2015 (mark all that apply)
	2009	2014	2015	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Provided funding to carry out an evaluation or hire an external evaluator (as part of the grant, or through other means)	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Provided training or technical assistance (by your staff, consultants or other means) to conduct rigorous evaluation	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Provided training or technical assistance (by your staff, consultants or other means) to support implementation of the program	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to support for grantees between 2014 and 2015, specify those factors below:

	Other major factor(s) that contributed to change between 2014 and 2015
A. Provided funding to carry out an evaluation or hire an external evaluator
B. Provided training or technical assistance to conduct rigorous evaluation
C. Provided training or technical assistance to support implementation of the program

2a. If your organization’s participation in SIF contributed to changes between 2014 and 2015 in the support your organization provides to grantees: How did SIF participation contribute to the change?

Evaluation

Evaluation is a systematic process to address such issues as the extent to which a program or intervention achieves its intended outcomes and impacts and how it can be improved. Organizations differ in their use of evaluations of programs that address community needs. In addition, the importance of evaluation as part of an organization's practice may change over time.

This section asks about your organization’s use of evaluation in 2015 and its evaluation resources and infrastructure. In addition, we are interested in changes your organization may have experienced over the 2014-2015 period, and the factors that contributed to those changes.

3. To what extent did your organization do the following regarding evaluation in 2015?
   
   For each thing that changed between 2014 and 2015, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2014 and 2015, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

CUSTOMIZED SURVEY WILL SHOW THE ORGANIZATION’S RESPONSES FOR 2009 AND 2014. THEN IT WILL ASK ABOUT 2015.

THE RESPONSES FOR YOUR ORGANIZATION FOR 2009 AND 2014 WERE:

	Extent to which organization did this in			Major factors that contributed to change between 2014 and 2015 (mark all that apply)
	2009	2014	2015	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Conducted rigorous evaluations of programs funded by your organization	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Used evaluation findings to improve programs funded by your organization	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization relating to evaluation between 2014 and 2015, specify those factors below:

	Other major factor(s) that contributed to change between 2014 and 2015
A. Conducted rigorous evaluations of programs funded by your organization
B. Used evaluation findings to improve programs funded by your organization
C. Used evaluation findings to demonstrate and communicate effectiveness of programs funded by your organization

3a. If your organization’s participation in SIF contributed to change in your organization's approach to evaluation: How did SIF participation contribute to the change?

4) Did/does your organization have the following in 2015?

	2015
Staff position(s) or group within your organization dedicated to evaluation	Yes/No
External evaluation partner(s) -- consultant(s) or organization(s) that provide your organization with evaluation services	Yes/No
Part of the organization's budget dedicated to evaluation	Yes/No

5) For this question, think about your organization's total evaluation budget in 2015 in comparison to 2014 (both in terms of dollars and as a percentage of the organization's total budget). Was your 2015 evaluation budget…

	Total evaluation budget in dollars	Evaluation budget as percentage of organization’s total budget for year
Substantially higher than the evaluation budget in 2014	[ ]	[ ]
Somewhat higher than the evaluation budget in 2014	[ ]	[ ]
About the same as the evaluation budget in 2014	[ ]	[ ]
Somewhat lower than the evaluation budget in 2014	[ ]	[ ]
Substantially lower than the evaluation budget in 2014	[ ]	[ ]

6) If you can access or estimate your organization's budget and staffing for evaluation in 2014 and 2015, please provide the following information:

	2014	2015
Total annual evaluation budget (in $) – for in-house evaluators or external partners
Annual evaluation budget as % of the total organization budget
Number of full time equivalent (FTE) staff with primary responsibility for evaluation

Scaling up of evidence-based programs: Increasing the impact of programs within the community or in other communities

A number of grantmakers are interested in scaling up programs that have shown evidence of effectiveness -- increasing the impact of a program within the community or expanding it to other communities or populations. This section asks about your organization’s involvement in efforts to scale programs.

7) To what extent did your organization do the following regarding scaling up of programs 2015?

For each thing that changed between 2014 and 2015, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2014 and 2015, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

CUSTOMIZED SURVEY WILL SHOW THE ORGANIZATION’S RESPONSES FOR 2009 AND 2014. THEN IT WILL ASK ABOUT 2015.

THE RESPONSES FOR YOUR ORGANIZATION FOR 2009 AND 2014 WERE:

	Extent to which organization did this in			Major factors that contributed to change between 2014 and 2015 (mark all that apply)
	2009	2014	2015	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Undertook efforts to scale up existing program(s) – i.e., to expand the program(s) within the community or to other communities or populations	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization’s approach to scaling up programs between 2014 and 2015, specify those factors below:

	Other major factor(s) that contributed to change between 2014 and 2015
A. Undertook efforts to scale up existing program(s)
B. Selected programs for scale-up based on rigorous evaluation that shows them to be effective

7a. If your organization’s participation in SIF contributed to change in your organization's approach to scaling up programs: How did SIF participation contribute to the change?

Collaboration to Address Community Needs

8) To what extent did your organization participate in collaborations to support implementation of programs in communities, in 2015?

For each thing that changed between 2014 and 2015, please indicate the factors that contributed to this change in the rightmost column. If there was no change between 2014 and 2015, mark “Not applicable (no change).” If you report that "other" major factors contributed to a change, please describe those factors below.

CUSTOMIZED SURVEY WILL SHOW THE ORGANIZATION’S RESPONSES FOR 2009 AND 2014. THEN IT WILL ASK ABOUT 2015.

THE RESPONSES FOR YOUR ORGANIZATION FOR 2009 AND 2014 WERE:

	Extent to which organization did this in			Major factors that contributed to change between 2014 and 2015 (mark all that apply)
	2009	2014	2015	Trends in the larger grantmaking world	Your organization’s participation in SIF	Specific requirements attached to the funding your organization receives	Your organization’s board/leadership directed organization to implement change	Other (please specify below)	Not Applicable (no change)
A. Participated in funding alliance(s) with other nonprofit sector organizations. (For example, co-funding programs through joint funding; providing or receiving matching funds; or other collaboration)	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
B. Participated in collaborations with other nonprofit organizations to share knowledge	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]
C. Collaborated with other organizations for purposes of advocacy – to advocate for or develop public support for programs or approaches to addressing social problems	2009 response will be inserted here	2014 response will be inserted here	Dropdown menu goes here (7-point scale from “Always to “Not at all”)	[ ]	[ ]	[ ]	[ ]	[ ]	[ ]

If you indicated that other major factors contributed to change in your organization between 2014 and 2015, specify those factors below:

	Other major factor(s) that contributed to change between 2014 and 2015
A. Participated in funding alliance(s) with other nonprofit sector organizations
B. Participated in collaborations with other nonprofit organizations to share knowledge
C. Collaborated with other organizations for purposes of advocacy

8a. If your organization’s participation in SIF contributed to change in your organization's collaboration: How did SIF participation contribute to the change?

Support Received by SIF Intermediaries

The preceding sections asked about change in your organization’s experience in different areas in the period between 2014 and 2015. Now think about technical assistance or similar support (e.g., coaching, facilitation, tools) your organization may have received during your SIF funding period from 2014 to 2015 to help you increase capacity or make changes. CNCS would like feedback from intermediaries to learn from intermediary experience and improve SIF services. 

Did your organization participate in SIF in the period from 2014 to 2015?

[] Yes

[] No – If no, skip to Q. 17

9) What kinds of support or resources have been especially helpful? Who provided them? (Please give examples)

10) Were there kinds of support or resources that have been less helpful to you? How could these be improved? (Please provide examples or suggestions)

11) Were there areas where you would have benefited from receiving more support or resources than you did? (Please provide examples)

Development of Capacity among Your Organization’s SIF Subgrantees

This section asks about your SIF subgrantees’ capacity to carry out different functions - at the time their SIF funding started and in 2015 - and the extent to which any change in their capacity were attributable to their participation in SIF.

In responding to these questions, please think about your SIF subgrantees overall or on average. If some subgrantees have experienced particularly great increases in capacity, or have faced particular challenges or have not increased capacity, you can mention those exceptions in examples.

12) Overall, how would you rate your SIF subgrantees' capacity to do the following, at the time their SIF funding started and in 2015? For areas where your SIF grantees’ capacity has increased, how much of that change is the result of their participation in SIF?

	Your SIF Subgrantees’ Capacity at the Time their SIF Funding Started
	Very Strong	Strong	Moderate	Weak	Very Weak
A. Implement the interventions they are carrying out in their communities
B. Design and conduct rigorous evaluations of their interventions
C. Make use of evaluation findings for program improvement
D. Raise matching funds for the intervention
E. Meet federal compliance requirements
F. Scale up the intervention (i.e., increase impact within community, or expand to other communities)
G. Share knowledge and best practices

	Your SIF Subgrantees’ Capacity in 2015
	Very Strong	Strong	Moderate	Weak	Very Weak
A. Implement the interventions they are carrying out in their communities
B. Design and conduct rigorous evaluations of their interventions
C. Make use of evaluation findings for program improvement
D. Raise matching funds for the intervention
E. Meet federal compliance requirements
F. Scale up the intervention (i.e., increase impact within community, or expand to other communities)
G. Share knowledge and best practices

	How much of the change in your subgrantees’ capacity has been a result of their participation in SIF?
	A substantial amount	Some	A little	None	Not applicable (no change)
A. Implement the interventions they are carrying out in their communities
B. Design and conduct rigorous evaluations of their interventions
C. Make use of evaluation findings for program improvement
D. Raise matching funds for the intervention
E. Meet federal compliance requirements
F. Scale up the intervention (i.e., increase impact within community, or expand to other communities)
G. Share knowledge and best practices

13) If there has been a change in your SIF subgrantees' capacity, please provide examples and describe how SIF has contributed to the changes. (These can include reduction in capacity as well as increase.)

14) If there are other areas where SIF subgrantee capacity has increased because of SIF, please describe:

15) If there has been a change in your SIF subgrantees' capacity, what other factors (other than SIF) have contributed to the change? (These can include reduction in capacity as well as increase.)

16) What have been some of the greatest challenges you have experienced in seeking to increase SIF subgrantee capacity, and what approaches have been most effective in addressing these challenges?

	Challenges in seeking to increase SIF subgrantee capacity	Approaches that have been most effective in addressing challenges
1.
2.
3.

Federal Funding

Federal Funding

19) Did your organization receive any federal government funding, other than SIF funding, in 2015?

( ) Yes

( ) No

Tiered Evidence Grant Programs.

Tiered evidence programs are programs where the funder awards grants based in part on the quality of past evidence and requires grantees to develop a higher level of evidence for the effectiveness of the programs based on rigorous evaluations.

The federal government funds several tiered evidence initiatives.  In addition to SIF, these include the Department of Education’s Investing in Innovation (i3) program, and the Workforce Innovation Fund (WIF) supported by the Departments of Labor, Education, and Health and Human Services.  

20) Based on your experience in SIF, how effective do you think tiered-evidence initiatives are in achieving such outcomes as building evidence in an area?

( ) Very effective

( ) Somewhat effective

( ) Not effective

20a. Please explain:

21) What do you think are the strengths or contributions of tiered-evidence initiatives?

22) What do you think are the problems or limitations of tiered-evidence initiatives?

Reflections on SIF experience

These last questions ask you for some final reflections on your organization’s experience with SIF.

23. Thinking about the successive years of your SIF funding – what were the major changes your organization experienced in each year of SIF funding and what were the reasons for those changes? (If you have not yet experienced the later years, please write NA for “not-applicable – have not yet experienced that year”)

	Changes Organization Experienced in Year	Major Reasons for Changes
First Year
Second Year
Third Year
Fourth Year
Fifth Year

24. What are the strengths/benefits of the SIF model compared with other programs you have participated in? Please provide examples.

25. What are the challenges/problems of the SIF model compared with other programs you have participated in? Please provide examples.

26. Thinking about your organization’s programs (other than SIF) in which you fund grantees to carry out programs in communities: has your SIF experience affected how you conduct those other programs?

( ) Yes

( ) No

( ) N/A – do not have any other programs in which organization funds grantees to carry out programs in communities

(If respondents select “Yes” to question 26) Please describe how your SIF experience has affected the way your organization conducts other grant programs.

27. Are there elements of SIF that your organization has sustained or will sustain over the longer term, after the completion of the period of SIF funding?

( ) Yes

( ) No

( ) Don’t know

(If respondents select “Yes” to question 27) Please describe the elements that will be sustained and how your organization will sustain them.

SIF elements that will be sustained by organization	How SIF elements will be sustained by organization

28. What advice would you give a federal agency that was considering use of an intermediary model similar to SIF?

28. What recommendations do you have for improving the SIF program?

Thank You!

Thank you for taking our survey.

Your response is very important to the Corporation for National and Community Service.

We will plan to contact you shortly to schedule a short telephone call to ask about documentation of changes related to your organization’s SIF participation. We are interested in documented evidence of change in such areas as organizational practice, policies and procedures, organizational structure, or changes in engagement with other organizations and the field. We will send you an email in advance and schedule a call for a time that is convenient for you.