Site Visits

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

2.2 RIF Site Visit Discussion Guide and Procedures
2.2.1 Proposed Site Visit Protocol
Table 2.2.1. Proposed Site Visit Locations
State

Award

Underserved
Area

RIF Grant
Type

RHED
Grants

KY

$300,000

Appalachia

Single Purpose

4

AZ

$1,999,177

Colonias

Comprehensive

3

LA

$300,000

MS Delta

Single Purpose

4

CA

$392,266

Tribal

Indian Economic
Development

1

Chippewa Cree Tribe

MT

$2,000,000

Tribal

Comprehensive

–

Oglala Sioux Housing
Authority

SD

$2,000,000

Tribal

Comprehensive

6

San Felipe Pueblo Housing
Authority

NM

$1,660,000

Tribal

Comprehensive

–

Colville Confederated Tribes

WA

$799,750

Tribal

Indian Economic
Development

–

Choctaw Nation of Oklahoma

OK

$776,294

Tribal

Indian Economic
Development

3

Organized Village of Kake

AK

$567,908

Tribal

Indian Economic
Development

–

Taos Pueblo

NM

$799,997

Tribal

Indian Economic
Development

–

Community Students Learning
Center

MS

$1,974,462

MS Delta

Comprehensive

–

Pathstone Corp.

PR

$2,000,000

Rural
Community

Comprehensive

–

Portable Practical Educational
Preparation

AZ

$1,557,840

Rural
Community

Comprehensive

4

Southern Bancorp Capital
Partners

AR

$1,998,621

Rural
Community

Comprehensive

–

Grantee
Preliminary Site Visits
Housing Development Alliance
(Hazard–Perry County)
Nogales Community
Development Co.
Macon Ridge Community
Development
Karuk Tribe of California

Primary Site Visits

The Proposed Site Visit locations in Table 2.2.1 were chosen based on two separate sets of
criteria. These criteria are based off research needs, contract constraints, and cost considerations.
We have included the selection criteria for both groups here.
Preliminary Site Visits: Selection Criteria
 RIF grantee (received recent award).

Page 1 of 7 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund






1046-000/ DU205NC-13-T-0007

Participated in both RIF and RHED programs.
Included all three types of RIF grantees (Single-Purpose, Comprehensive, and Tribal).
Included grantees from all four of RHED’s underserved communities.
Received multiple awards.

Primary Site Visits and Constraints: Selection Criteria
 Seven of the eleven sites must be Tribal grantees.
 Recipients of larger grants will be preferred, specifically Indian Economic Development
and Entrepreneurship grants and Comprehensive grants.
 When possible, extremely remote sites that will strain the travel budget will be avoided.
We have already completed the four preliminary site visits to test our data collection instruments
and gather some initial data on rural housing and community development. The procedures and
protocols described in this section are based on these early visits and have been successful. Table
2.2.1 includes all our proposed site visit locations based on the selection criteria.
We believe that this set of site visit locations provides the best opportunity to answer the
evaluation’s research questions while balancing the constraints of a finite travel budget and a
requirement to visit tribal grantees. The protocols and discussion guide below have been updated
based on our experiences with the Preliminary Site Visits and describe our approach to
conducting the site visits.
Site Visit Protocols
Prior to the visit:


Confirm proposed site visit list with PD&R.



Contact RIF Program Office and ask for introductory email to both grantee and HUD
Field Office.



Conduct introductory call with Program Office, Econometrica, Field Office, and Grantee.
Set tentative week for site visit.



Send Discussion Guide and other documents to grantee. Confirm dates and schedule team
members for trip. Ask for recommendations for partners and other interviews.



Contact grantees 2 to 3 business days prior to site visit to confirm details.

During the Visit:


Morning, Day 1: Introduce site visit team to primary grantee contact. Have them
introduce us to the other grantee staff. Identify which staff it would be profitable to
interview later. Conduct initial background and status interview (~1.5 hours).



Mid-Day, Day 1: Conduct a tour of various project sites, not just RHED/RIF projects.
Tour local projects, interview project site coordinators, clients, contractors and others.



Afternoon, Day 1: Conduct interviews with other financial and technical staff. Interview
local partners, if available.

Page 2 of 7 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007



Morning, Day 2: Conduct follow-up interview with RIF Program Coordinator. Conduct
any remaining staff interview.



Mid-Day, Day 2: Specifically tour RIF project site, if not toured on previous day. Talk to
clients, contractors, etc.



Afternoon, Day 2: Conduct final interviews, gather requested documents, and hold
debriefing session with Executive Director/RIF Program Coordinator.

Following the Visit:


Send “thank you” communications to grantee staff and other respondents.



Complete site visit report within 15 business days, focusing on leveraging/partnerships,
capacity building, and program outcomes and impact. Forward to HUD.



Summarize data from site visit for report. Organize any photos taken.

Page 3 of 7 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

Primary Site Visit Discussion Guide
Evaluation of the Rural Innovation Fund (RIF)
Preliminary Site Visit Discussion Guide
Objective: The Econometrica project team is conducting an assessment of RIF under contract to
HUD’s Office of Policy Development and Research (PD&R). The purpose of this guide is to
provide a structure for the discussions that will take place during the team’s preliminary site
visits. The objective of the site visits is to obtain “on the ground” information about the past
performance of RIF grantees and identify realistic data sources that can be used to measure the
impact of RIF-supported activities. The input from these preliminary site visits will be an
important information source in the development of the study’s data collection protocols,
particularly the structured survey instrument.
Eleven (11) site visits will take place at a selection of grantee sites that will include most of the
RIF grantees that received either the larger comprehensive grants or the Tribes that received the
Economic Development and Entrepreneurship grants. Some of these organizations also received
Rural Housing and Economic Development (RHED) grant funds when that program was actively
providing grants. Part of the analysis will be about differences between the two grant programs.
Two senior members of the Econometrica project team will conduct the site visits. Specific
agendas for each visit will be developed via discussions with grantee representatives in advance
of the visits. We anticipate that each visit will take about a day and a half, although the study
teams will be prepared to stay longer if necessary to obtain significant input or data. The site visit
team will attempt to identify stakeholders that are familiar with the grantee and its work prior to
scheduling the visits; the team will attempt to meet with these partners as well.
The site visit teams will submit site visit reports to PD&R within 15 business days of each visit.
Following are overviews of the major discussion topics that will be covered in each site visit.

Discussion Topic 1: Current and Past Administration of RIF/RHED Grants


Introductions with lead contacts at the grantee organization.



Confirmation/modification of pre-established agenda.



Overview of current administration/management of grantee organization. Has that
management structure remained constant for all of your RIF and RHED grants?



What are the major goals of the grantee organization? What does it see as its mission?
What has been the organization’s major type of activities, aside from those funded by
RIF grants?



How have the activities funded by RIF been integrated into the organization’s mission
and goals?



What specific community problems did your organization believe could be solved
effectively by using the RIF grants?
Page 4 of 7 Pages

Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007



Were there any unanticipated issues or barriers that affected your grant activities? If yes,
what were they and what strategies did you use to address them? In retrospect, do you
think those strategies were effective?



Which community stakeholders outside of the current grantee staff could provide a good
perspective on your RIF and RHED grant activities? These stakeholders could include
residents, government representatives, nonprofit executives, or others familiar with the
grantee.

Discussion Topic 2: Leveraging Funds and Resources


Did you assume that you would need to leverage funds and/or resources to successfully
complete your proposed RIF projects? If yes, what were the sources of these leveraged
funds or resources?



In your opinion, how critical was winning a RIF award to your ability to obtain leveraged
funds or resources? For example, could you have used sources of funds other than RIF to
obtain the leveraged funds/resources?



Did you obtain the ratio of leveraged funds/resources to your grant amounts that you
originally anticipated? What were the factors that influenced these results?



How did the size of the grant affect the amount of funds leveraged?



Is your organization still benefitting from the leveraging process that was associated with
your RIF grants, and if so, how?

Discussion Topic 3: Capacity and Partners


Has your financial strength—capital and operating—improved due to the RIF grant? If
yes, what advantages have come from this improved position?



Can you cite specific examples of how the RIF grant has helped you because of increased
financial strength? Increased organizational strength?



How have your organization’s planning skills been enhanced by the RIF grant?



Has your staff size or skill levels increased because of the grant? Do you have any new
procedures or systems in place since obtaining the grant?



Have you added any new board members since obtaining the grant?



Did you see partnering as an essential component of the RIF-funded programs? If yes,
who were the partners targeted and what roles were they to play?



What new partners have you developed since the RIF grant? In what way do they help
you meet your objectives? Did the size of the RIF grant help in developing these
partnerships?



Has the organization become more self-sufficient (earned revenues as a portion of
expenses)?
Page 5 of 7 Pages

Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund



1046-000/ DU205NC-13-T-0007

Has there been any area in which your organization has needed Technical Assistance to
effectively implement your RIF/RHED grant?

Discussion Topic 4: Impacts and Measurement


What were the outputs from the RIF grant, including housing units, jobs, and businesses?
Were there any spillover effects?



How do these outputs relate to your organization’s mission and goals?



What indicators/metrics did you use to measure progress toward your objectives? How do
you collect and organize data on outputs and outcomes? What are your sources of data?



Do you have short- and long-term objectives that you measure progress toward?



Based on your experience, what do you believe are the best measures of your programs’
impact? How would you prefer your performance to be measured?



After the start of your RIF/RHED projects, did you identify some new data items that you
thought were particularly useful in measuring progress and impact? If yes, please specify
these items and explain why you have found them useful.



Are there any data that you have not been able to obtain as yet that you think would be
particularly useful in measuring the impact of one or more of your projects? If yes, please
specify what these data are, explain why you think they are particularly informative, and
indicate the barriers to obtaining the data.



Should we consult any national or State data sources when evaluating the impact of the
RIF/RHED grants on your community? Examples would be U.S. Census data, Home
Mortgage Disclosure Act data, and Bureau of Labor Statistics data.

Discussion Topic 5: Document Requests
 Many non-profit organizations are required to have annual audits conducted. We often
find that these documents contain a wealth of information that is helpful for our
evaluation of the overall RIF Program. Do you have copies of your annual audits, and, if
yes, may we obtain copies of them? This information will not be provided to HUD
directly and will not be identified with your organization.
 Has your organization completed a strategic plan or other document laying our
organizational goals? If yes, may we have a copy of it for our report?

Discussion Topic 6: Topics for Tribal Grantees (Optional)


What part of the Tribal governing authority was primarily responsible for deciding to
apply for the RIF grant? Was it a Department, such as Housing, or was it the Tribal
Council or Executive Authority?

Page 6 of 7 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund















1046-000/ DU205NC-13-T-0007

Was the RIF grant a new initiative for the Tribe or did it build on some previous grant
awards (such as the RHED)?
Were the activities proposed in your RIF application part of a larger Tribal initiative or
was the grant intended to focus on a single objective?
Was the flexibility provided by the RIF grant attractive compared to restrictions placed
on some Native American programs such as those of the Indian Health Service (IHS) and
the Bureau of Indian Affairs (BIA)?
Which Tribal Department has had primary responsibility for managing/implementing the
RIF grant? Why was that Department selected to lead the project?
Are the activities funded under the RIF grant intended to enhance other commercial
activities of the Tribe such as a casino or retail outlet?
Have you worked with any non-Tribal partners in implementing the RIF grant? If yes,
how successful has the partnership been?
Are the project benefits available to both tribal and non-tribal members?
Has the Tribe been able to establish an effective working relationship with HUD
representatives responsible for monitoring the RIF grant? Have there been any problems
in establishing a government-to-government relationship?
Does the Tribe have in place all the necessary resources such as accounting systems and
IT capabilities to effectively manage the RIF-funded project?
Has there been any area in which the Tribe has needed Technical Assistance to
effectively implement the grant?
What data do you think will most clearly indicate the positive impact of the RIF-funded
project (e.g., higher employment on the reservation, enhanced cash flow in business
enterprises, improvement in health indicators)?

Page 7 of 7 Pages
Econometrica, Inc.

January 5, 2014