7 SIF Intermediary Evidence Interview

SIF Grantee Telephone Follow-up Interview: Evidence of Change

Email Text (send after survey, before telephone interview)

In the SIF National Assessment survey you filled out recently, you provided information on changes in your organization between 2009 and 2014 [for 2016 round, change to “between 2014 and 2015”] and the effects of SIF participation and other factors on change. As we indicated in the survey, we will telephone you to discuss objective evidence you have for change in your organization related to your participation in SIF.

The list below provides illustrative examples of some kinds of evidence of change. It may be helpful to think about these or other examples before we call you. When we call you, we will ask you about these or other kinds of evidence for change, so we can understand the change better and, if possible, obtain examples of evidence of change.

• Change in organizational practice, such as:

  • Increased use of competitive solicitations or requirements for evaluation as a condition of funding – for example, recent solicitations with explicit language regarding the solicitation process or guidance memos directing the use of rigorous evaluation methods.

  • Changes in staff hiring, training or roles and responsibilities – for example, hiring evaluation staff with advanced methodological training, in-house training to increase program staff’s capacity to use evaluation findings for program improvement, or requirements for program staff to monitor grantee compliance with federal regulations

• Change in organization’s written policies and procedures

• Change in organizational structure, such as

  • Creation of a separate evaluation unit within the organization

  • Creation of a unit within the financial or accounting department responsible for compliance with federal regulations

• New or changed engagement with the field, such as:

  • Establishment of MOUs to co-fund programs in communities

  • Membership/participation in associations or networks in the field (e.g., American Achieves, Social Innovation Exchange, Innovation Network, or others)

  • Publication or presentations at professional conferences

We look forward to talking with you and appreciate your time and contribution to this study.

Interview Guide for the Telephone Follow-Up

Introduction

Good afternoon. We appreciate you taking the time to talk with us today. My name is _________, with ICF International, and I am joined here by my colleague _________.

In the SIF National Assessment survey you filled out recently, you provided information on changes in your organization between 2009 and 2014 [for 2016 round, change to “between 2014 and 2015”] and the effects of SIF participation and other factors on any change.

The purpose of this telephone call is to discuss objective evidence you may have for change. As we noted in the email we sent, objective evidence may include such things as changes in organizational practice, policies and procedures, organizational structure, or greater engagement with other organizations and the field.

This interview will be fairly open-ended. ________ will be taking notes to ensure we capture the discussion accurately, and we will also be recording the call in case we need to refer back to the information. We may include some of the specific examples and insights that you provide in the National Assessment Report. If we do, we’ll ask you to review that information to ensure that we have represented your organization’s information accurately.

• Do you have any questions about the purpose this interview before we get started?

• Do you have any questions about how the information you share today will be used?

Interview Questions

Can you tell us about the kinds of objective evidence your organization has for change that has come about through participation in SIF? [If needed – see prompts below]

1. What kind of documentation do you have for change?

2. What areas are the documented changes in – for instance, selection of grantees, evaluation, collaboration, scaling up of effective programs…..?

3. When was the change made? What was the process for making the change?

4. What is the evidence that the change was attributable to SIF participation? (rather than other factors or more general change)

5. Are there kinds of evidence you have found especially useful for documenting and communicating about change to funders or other stakeholders?

6. Which evidence of change would you be willing to share? What examples do you think might provide useful evidence for the National Assessment and CNCS? Can you provide us with specific examples or documentation?

Optional (ask if there is time):

7. Are there other topics or issues about your organization’s experience with SIF that you think were not fully captured in your responses to the survey? (If yes – discuss)

8. Are there other things about SIF that it would be useful for us to know for the National Assessment?

Wrap-Up

Thank you. This has provided valuable information on evidence for organizational change. We appreciate your time and contribution to this study. If you have anything else to add or have any questions, please do not hesitate to contact me at ____________________.

__________________________________________________________________________________

Prompts (Interviewer use these if not mentioned in interview – these are examples of kinds of evidence for change)

• Change in organizational practice, such as:

  • Increased use of competitive solicitations or requirements for evaluation as a condition of funding – for example, recent solicitations with explicit language regarding the solicitation process or guidance memos directing the use of rigorous evaluation methods.

  • Changes in staff hiring, training or roles and responsibilities – for example, hiring evaluation staff with advanced methodological training, in-house training to increase program staff’s capacity to use evaluation findings for program improvement, or requirements for program staff to monitor grantee compliance with federal regulations

• Change in organization’s written policies and procedures

• Change in organizational structure, such as

  • Creation of a separate evaluation unit within the organization

  • Creation of a unit within the financial or accounting department responsible for compliance with federal regulations

• New or changed engagement with the field, such as:

  • Establishment of MOUs to co-fund programs in communities

  • Membership/participation in associations or networks in the field (e.g., American Achieves, Social Innovation Exchange, Innovation Network, or others)

  • Publication or presentations at professional conferences

3