Site Visits

Evaluation of the Rural Innovation Fund 1046-000/ DU205NC-13-T-0007

2.2 RIF Site Visit Discussion Guide and Procedures

2.2.1 Proposed Site Visit Protocol

Table 2.2.1. Proposed Site Visit Locations

Grantee	State	Award	Underserved Area	RIF Grant Type	RHED Grants
Preliminary Site Visits
Housing Development Alliance (Hazard–Perry County)	KY	$300,000	Appalachia	Single Purpose	4
Nogales Community Development Co.	AZ	$1,999,177	Colonias	Comprehensive	3
Macon Ridge Community Development	LA	$300,000	MS Delta	Single Purpose	4
Karuk Tribe of California	CA	$392,266	Tribal	Indian Economic Development	1
Primary Site Visits
Chippewa Cree Tribe	MT	$2,000,000	Tribal	Comprehensive	–
Oglala Sioux Housing Authority	SD	$2,000,000	Tribal	Comprehensive	6
San Felipe Pueblo Housing Authority	NM	$1,660,000	Tribal	Comprehensive	–
Colville Confederated Tribes	WA	$799,750	Tribal	Indian Economic Development	–
Choctaw Nation of Oklahoma	OK	$776,294	Tribal	Indian Economic Development	3
Organized Village of Kake	AK	$567,908	Tribal	Indian Economic Development	–
Taos Pueblo	NM	$799,997	Tribal	Indian Economic Development	–
Community Students Learning Center	MS	$1,974,462	MS Delta	Comprehensive	–
Pathstone Corp.	PR	$2,000,000	Rural Community	Comprehensive	–
Portable Practical Educational Preparation	AZ	$1,557,840	Rural Community	Comprehensive	4
Southern Bancorp Capital Partners	AR	$1,998,621	Rural Community	Comprehensive	–

The Proposed Site Visit locations in Table 2.2.1 were chosen based on two separate sets of criteria. These criteria are based off research needs, contract constraints, and cost considerations. We have included the selection criteria for both groups here.

Preliminary Site Visits: Selection Criteria

• RIF grantee (received recent award).

• Participated in both RIF and RHED programs.

• Included all three types of RIF grantees (Single-Purpose, Comprehensive, and Tribal).

• Included grantees from all four of RHED’s underserved communities.

• Received multiple awards.

Primary Site Visits and Constraints: Selection Criteria

• Seven of the eleven sites must be Tribal grantees.

• Recipients of larger grants will be preferred, specifically Indian Economic Development and Entrepreneurship grants and Comprehensive grants.

• When possible, extremely remote sites that will strain the travel budget will be avoided.

We have already completed the four preliminary site visits to test our data collection instruments and gather some initial data on rural housing and community development. The procedures and protocols described in this section are based on these early visits and have been successful. Table 2.2.1 includes all our proposed site visit locations based on the selection criteria.

We believe that this set of site visit locations provides the best opportunity to answer the evaluation’s research questions while balancing the constraints of a finite travel budget and a requirement to visit tribal grantees. The protocols and discussion guide below have been updated based on our experiences with the Preliminary Site Visits and describe our approach to conducting the site visits.

Site Visit Protocols

Prior to the visit:

• Confirm proposed site visit list with PD&R.

• Contact RIF Program Office and ask for introductory email to both grantee and HUD Field Office.

• Conduct introductory call with Program Office, Econometrica, Field Office, and Grantee. Set tentative week for site visit.

• Send Discussion Guide and other documents to grantee. Confirm dates and schedule team members for trip. Ask for recommendations for partners and other interviews.

• Contact grantees 2 to 3 business days prior to site visit to confirm details.

During the Visit:

• Morning, Day 1: Introduce site visit team to primary grantee contact. Have them introduce us to the other grantee staff. Identify which staff it would be profitable to interview later. Conduct initial background and status interview (~1.5 hours).

• Mid-Day, Day 1: Conduct a tour of various project sites, not just RHED/RIF projects. Tour local projects, interview project site coordinators, clients, contractors and others.

• Afternoon, Day 1: Conduct interviews with other financial and technical staff. Interview local partners, if available.

• Morning, Day 2: Conduct follow-up interview with RIF Program Coordinator. Conduct any remaining staff interview.

• Mid-Day, Day 2: Specifically tour RIF project site, if not toured on previous day. Talk to clients, contractors, etc.

• Afternoon, Day 2: Conduct final interviews, gather requested documents, and hold debriefing session with Executive Director/RIF Program Coordinator.

Following the Visit:

• Send “thank you” communications to grantee staff and other respondents.

• Complete site visit report within 15 business days, focusing on leveraging/partnerships, capacity building, and program outcomes and impact. Forward to HUD.

• Summarize data from site visit for report. Organize any photos taken.

Primary Site Visit Discussion Guide

Evaluation of the Rural Innovation Fund (RIF)

Preliminary Site Visit Discussion Guide

Objective: The Econometrica project team is conducting an assessment of RIF under contract to HUD’s Office of Policy Development and Research (PD&R). The purpose of this guide is to provide a structure for the discussions that will take place during the team’s preliminary site visits. The objective of the site visits is to obtain “on the ground” information about the past performance of RIF grantees and identify realistic data sources that can be used to measure the impact of RIF-supported activities. The input from these preliminary site visits will be an important information source in the development of the study’s data collection protocols, particularly the structured survey instrument.

Eleven (11) site visits will take place at a selection of grantee sites that will include most of the RIF grantees that received either the larger comprehensive grants or the Tribes that received the Economic Development and Entrepreneurship grants. Some of these organizations also received Rural Housing and Economic Development (RHED) grant funds when that program was actively providing grants. Part of the analysis will be about differences between the two grant programs.

Two senior members of the Econometrica project team will conduct the site visits. Specific agendas for each visit will be developed via discussions with grantee representatives in advance of the visits. We anticipate that each visit will take about a day and a half, although the study teams will be prepared to stay longer if necessary to obtain significant input or data. The site visit team will attempt to identify stakeholders that are familiar with the grantee and its work prior to scheduling the visits; the team will attempt to meet with these partners as well.

The site visit teams will submit site visit reports to PD&R within 15 business days of each visit.

Following are overviews of the major discussion topics that will be covered in each site visit.

Discussion Topic 1: Current and Past Administration of RIF/RHED Grants

• Introductions with lead contacts at the grantee organization.

• Confirmation/modification of pre-established agenda.

• Overview of current administration/management of grantee organization. Has that management structure remained constant for all of your RIF and RHED grants?

• What are the major goals of the grantee organization? What does it see as its mission? What has been the organization’s major type of activities, aside from those funded by RIF grants?

• How have the activities funded by RIF been integrated into the organization’s mission and goals?

• What specific community problems did your organization believe could be solved effectively by using the RIF grants?

• Were there any unanticipated issues or barriers that affected your grant activities? If yes, what were they and what strategies did you use to address them? In retrospect, do you think those strategies were effective?

• Which community stakeholders outside of the current grantee staff could provide a good perspective on your RIF and RHED grant activities? These stakeholders could include residents, government representatives, nonprofit executives, or others familiar with the grantee.

Discussion Topic 2: Leveraging Funds and Resources

• Did you assume that you would need to leverage funds and/or resources to successfully complete your proposed RIF projects? If yes, what were the sources of these leveraged funds or resources?

• In your opinion, how critical was winning a RIF award to your ability to obtain leveraged funds or resources? For example, could you have used sources of funds other than RIF to obtain the leveraged funds/resources?

• Did you obtain the ratio of leveraged funds/resources to your grant amounts that you originally anticipated? What were the factors that influenced these results?

• How did the size of the grant affect the amount of funds leveraged?

• Is your organization still benefitting from the leveraging process that was associated with your RIF grants, and if so, how?

Discussion Topic 3: Capacity and Partners

• Has your financial strength—capital and operating—improved due to the RIF grant? If yes, what advantages have come from this improved position?

• Can you cite specific examples of how the RIF grant has helped you because of increased financial strength? Increased organizational strength?

• How have your organization’s planning skills been enhanced by the RIF grant?

• Has your staff size or skill levels increased because of the grant? Do you have any new procedures or systems in place since obtaining the grant?

• Have you added any new board members since obtaining the grant?

• Did you see partnering as an essential component of the RIF-funded programs? If yes, who were the partners targeted and what roles were they to play?

• What new partners have you developed since the RIF grant? In what way do they help you meet your objectives? Did the size of the RIF grant help in developing these partnerships?

• Has the organization become more self-sufficient (earned revenues as a portion of expenses)?

• Has there been any area in which your organization has needed Technical Assistance to effectively implement your RIF/RHED grant?

Discussion Topic 4: Impacts and Measurement

• What were the outputs from the RIF grant, including housing units, jobs, and businesses? Were there any spillover effects?

• How do these outputs relate to your organization’s mission and goals?

• What indicators/metrics did you use to measure progress toward your objectives? How do you collect and organize data on outputs and outcomes? What are your sources of data?

• Do you have short- and long-term objectives that you measure progress toward?

• Based on your experience, what do you believe are the best measures of your programs’ impact? How would you prefer your performance to be measured?

• After the start of your RIF/RHED projects, did you identify some new data items that you thought were particularly useful in measuring progress and impact? If yes, please specify these items and explain why you have found them useful.

• Are there any data that you have not been able to obtain as yet that you think would be particularly useful in measuring the impact of one or more of your projects? If yes, please specify what these data are, explain why you think they are particularly informative, and indicate the barriers to obtaining the data.

• Should we consult any national or State data sources when evaluating the impact of the RIF/RHED grants on your community? Examples would be U.S. Census data, Home Mortgage Disclosure Act data, and Bureau of Labor Statistics data.

Discussion Topic 5: Document Requests

• Many non-profit organizations are required to have annual audits conducted. We often find that these documents contain a wealth of information that is helpful for our evaluation of the overall RIF Program. Do you have copies of your annual audits, and, if yes, may we obtain copies of them? This information will not be provided to HUD directly and will not be identified with your organization.

• Has your organization completed a strategic plan or other document laying our organizational goals? If yes, may we have a copy of it for our report?

Discussion Topic 6: Topics for Tribal Grantees (Optional)

• What part of the Tribal governing authority was primarily responsible for deciding to apply for the RIF grant? Was it a Department, such as Housing, or was it the Tribal Council or Executive Authority?

• Was the RIF grant a new initiative for the Tribe or did it build on some previous grant awards (such as the RHED)?

• Were the activities proposed in your RIF application part of a larger Tribal initiative or was the grant intended to focus on a single objective?

• Was the flexibility provided by the RIF grant attractive compared to restrictions placed on some Native American programs such as those of the Indian Health Service (IHS) and the Bureau of Indian Affairs (BIA)?

• Which Tribal Department has had primary responsibility for managing/implementing the RIF grant? Why was that Department selected to lead the project?

• Are the activities funded under the RIF grant intended to enhance other commercial activities of the Tribe such as a casino or retail outlet?

• Have you worked with any non-Tribal partners in implementing the RIF grant? If yes, how successful has the partnership been?

• Are the project benefits available to both tribal and non-tribal members?

• Has the Tribe been able to establish an effective working relationship with HUD representatives responsible for monitoring the RIF grant? Have there been any problems in establishing a government-to-government relationship?

• Does the Tribe have in place all the necessary resources such as accounting systems and IT capabilities to effectively manage the RIF-funded project?

• Has there been any area in which the Tribe has needed Technical Assistance to effectively implement the grant?

• What data do you think will most clearly indicate the positive impact of the RIF-funded project (e.g., higher employment on the reservation, enhanced cash flow in business enterprises, improvement in health indicators)?

Page 7 of 7 Pages

Econometrica, Inc. May 27, 2014