RHED survey

Section 2: Rural Innovation Fund (RIF) Evaluation Data
Collection Tools
This section provides a facsimile of the RIF Telephone Survey Instrument, the RHED Telephone
Survey Instrument, and the Site Visit Discussion Guide, as well as the relevant material that will
precede or accompany the survey. We will conduct this survey with all 51 RIF grantees who
received funding from the FY 2010 NOFA and approximately 50 RHED grantees who received
awards between 2005 and 2009.

2.1 RIF/RHED Telephone Survey Instruments and Procedures
2.1.1 Proposed Pre-Survey Procedures & Sample Selection Criteria
For the RIF telephone surveys, it will be necessary to identify a “primary respondent” from each
grantee who will respond to the survey. We will begin the process by identifying the grantee
coordinator or administrator from the administrative files, grant applications, and more recent
reports from the grantee. These files and access to the reports have already been provided the
RIF program and we will develop a list of these individuals prior to beginning the telephone
interviews. We will then ask the RIF Program Office to initiate contact with the grantee to
schedule an appropriate time to hold the interview. The local HUD Field Office will also be
notified at this time.
For the RHED telephone survey, we will use the selection criteria spelled out below and in
Section 4.B. to identify a purposive sample of recent RHED grant recipients. Despite the
difficulties in comparing the responses between interviewing the universe of grant recipients –
the RIF interview sample – and a purposive sample of grantees – the RHED interview sample –
we believe this approach best compensates for the lack of consistent administrative program
data, particularly in the RHED program. While some small amount of administrative data are
consistently available across grantees and programs, the RHED administrative data are
disorganized and unreliable. For example, the grant award amount frequently does not
correspond between the Program Office’s electronic records and hard copies in the
administrative files. In another case, the Program Office has been unable to provide us with
contact information for current or former grantees. This discrepancy was one of the initial
findings of our study team and has heavily influenced the eventual research and data collection
approach we propose. Our methods were selected with this limitation – and contractual
requirements – in mind. More information on these issues can be found below in Section 4.B.
Question 1.
As mentioned in Section 1.3.2, there are 141 unique RHED grantees who received awards
between 2005 and 2009. After removing the 23 RHED grantees from that period who also
received a RIF grant in order distinguish between the two programs, 118 recent RHED grantees
remain to select our RHED sample from using the criteria below.
RHED Telephone Interviews: Selection Criteria


Received a RHED grant after 2005 (310 grants).






Received multiple RHED awards (141 unique, repeat RHED grantees).
Did not receive a RIF grant (118 unique, repeat grantees).
Sample will include similar numbers of the four disadvantaged communities.
Total number of grantees interviewed will be no less than 50 and no more than 75. Final
numbers will depend on staff availability and organizational permanence.

After we select our RHED survey sample, we will ask the RHED Program Office and the HUD
Field Office to establish contact with the grantee. From this communication and a review of the
program files, we will determine which grantee staff member had or has primary responsibility
for managing the RHED grants. We will then contact the staff member directly and schedule an
appropriate time for the interview at the convenience of the grantee.
We will attempt to complete most of the logistical preparation while awaiting OMB clearance.
After we receive OMB approval, we will notify the RIF and RHED Program Offices that it is
time to contact the grantees and introduce the Econometrica team. Econometrica will then
schedule the interviews directly with the grantees. In Section 2.2, we have provided an email
template introduction to the RIF Evaluation Project, the Econometrica team, and the telephone
survey.
2.1.2. Proposed Telephone Survey Email Invitation
Dear [insert name of RIF/RHED project contact here]:
The U.S. Department of Housing and Urban Development (HUD) is seeking your help. HUD has
contracted with Econometrica, Inc.—a research firm located in Bethesda, MD—to conduct a
study of the Rural Innovation Fund (RIF) program. The purpose of the study is to explore how
the larger grant amounts available through the RIF have changed program activities, impacts,
and outcomes relative to the RHED.
As part of their study, Econometrica will conduct a series of telephone interviews with
representatives from all RIF grantees and a sample of RHED grantees. This information is
necessary to evaluate the impact of the RIF program relative to the RHED program and will help
HUD develop policy approaches to rural housing and economic development issues in the future.
We have identified [insert name of RIF/RHED project contact here] as the Project Coordinator
for [insert name of grantee organization]’s [insert RIF/RHED grant number]. If this information
is incorrect or out of date, please respond with the person currently responsible for [insert
RIF/RHED grant number]. If no available staff members have actively managed this grant, we
would like to interview the current Executive Director of [insert name of grantee organization].1

1

Since all RIF projects are still active, we believe it is unlikely that we will need to include this sentence in the
introductory email to RIF grantees. However, many RHED grantees may have closed their projects years ago and
high staff turnover in rural organizations means it is likely that we will have to include this or a similar sentence in
the introductory email to RHED grantees.

This email serves as an introduction to key project staff including the HUD Representative,
Ndeye Jackson, and the Econometrica Project Lead, James Hedrick. Econometrica staff will be
working with you directly to schedule a time to conduct the telephone interview. They will be
following up with you to determine your availability within the next 2 weeks. Please work with
them to establish a time to conduct the interview.
The interview should take approximately 45 minutes to complete. Due to the open-ended nature
of some of the questions, the interview may last up to 1 hour. Please allow for extra time when
scheduling.
If you have any questions about the telephone survey or the overall evaluation, please do not
hesitate to contact Ndeye Jackson, GTR/COTR, in HUD’s Office of Policy Development and
Research (PD&R) at (202) 402-5737. She will be pleased to talk with you.
Thank you for your hard work and we look forward to your participation in this research effort.
Sincerely,

[Aaron Taylor or Jackie Williams]
[Position and Office]
Department of Housing and Urban Development

2.1.3 Proposed Pre-Survey Instructions Script
This survey concerns the RIF/RHED grant for project number ________________, originally
approved in _______ (fiscal year); known as the ______________________________ (project
name); with the following purpose:
______________________________________________________________________________
______________________________________________________________________________
____________________________________________________
__.
The purpose of this evaluation is to explore how the larger grant amounts available through the
RIF have changed program activities, impacts, and outcomes relative to the RHED.
First and foremost, this is not an audit or a monitoring exercise. Econometrica is performing a
programmatic evaluation of the RIF and RHED programs. We are interested in hearing from you
about how you operated your grants, what partners you engaged, what resources you leveraged,
how the grants affected your organization’s capacity, and the overall impact of the project on the
community. No information you provide will be used to evaluate your organization specifically.
Additionally, your responses will be anonymous and will not be attached to you specifically or to
your organization. We may quote some passages or responses to open-ended questions, but we
will not identify the source of the comment.
Please answer honestly and fully to the best of your knowledge. If you are unsure of any question
or believe that someone else would be better able to answer any question, please let us know and
we will make a note of it in your response.
Thank you for your participation. If you have any questions about the survey or any concerns
you would like to express to HUD, you may contact Ndeye Jackson, GTR/COTR, in HUD’s
Office of Policy Development and Research (PD&R) at (202) 402-5737.
If you are ready, we can begin.

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

2.1.4. Proposed Survey Instrument for the Evaluation of the Rural Innovation Fund
Organization of the Survey
I. General Information
II. Grant Use and Capacity Building
III. Grant Application and Management
IV. Leveraging
V. Partnering-Linkages
VI. Program Outcomes and Impact
VII. Conclusion

Page 5 of 11 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

Rural Housing and Economic Development (RHED)
Grantee Telephone Interview Instrument

Section I: General Information
Grantee Entity: (Organizational entity submitting grant application and winning grant award.)
Respondent: (Organizational official currently responsible for managing RHED grant activities
or follow-up to grant activities. Additional staff may participate in interview if requested.
Confirm positions and names at beginning of interview.)
Date of Interview: (mm/dd/yy)
Project Team Interviewer: ________________________________________________________

Section II: Grant Use and Capacity Building
1. For any RHED grant, were the grant funds used in any manner that differed from your
application? Please describe differences.
________________________________________________________________________
2. Have the number of your FTEs changed due to the funding you received via any RHED
grant?
Yes

No

If yes, to what extent do you think any RHED grant contributed to your organizational
growth?
a.
b.
c.
d.

Growth attributed almost solely to RHED.
Growth attributed largely to RHED.
Growth somewhat attributed to RHED.
RHED one of many factors leading to growth.

If staff has decreased since the completion of your RHED grant, what factors led to the
decrease?
________________________________________________________________________
3. An often-used measure of an organization’s basic financial strength is assets divided by
liabilities. How would you characterize your financial strength today compared to when
you received your first RHED grant?
a. Financial strength is better than when grant was awarded.
b. Financial strength is about the same as when grant was awarded.
c. Financial strength is worse than when grant was awarded.
4. How would you characterize the impact of your RHED grants on your organization’s
financial strength? Did any RHED grant help your organization improve your financial
strength in any way? If so, how?
________________________________________________________________________
_______________________________________________________________________
Page 6 of 11 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

5. Did your RHED grant(s) help your organization moved into areas of activity where you
had not previously been active?
Yes

No

If yes, what were the new areas of activity?
________________________________________________________________________
If yes, how did the RHED program help your organization to move into this/these new
area(s)?
________________________________________________________________________
________________________________________________________________________

6. In your opinion, how important were your RHED grant(s) in enabling your organization
to address the issues – such as high unemployment, low education levels, etc. – that your
organization listed in your grant application?
________________________________________________________________________
________________________________________________________________________

7. Did your organization consider any other grant sources to address the issues you included
in your application?
Yes

No

Unsure

If yes, what other sources?
________________________________________________________________________
________________________________________________________________________

Section III: Grant Application and Management
1. How would you characterize the current status of your RHED-funded project(s)?
________________________________________________________________________
________________________________________________________________________
2. How would you characterize the status of your RHED-funded project(s) relative to the
original schedule from your application?
a.
b.
c.
d.
e.

Completed ahead of schedule.
Completed on schedule.
Still in progress.
Completed behind schedule.
Experienced significant delays or project was not completed.

Page 7 of 11 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

If any of your RHED project(s) were not completed as scheduled or failed to be
completed, what barriers did you encounter when implementing the project?
________________________________________________________________________
________________________________________________________________________
3. How would you characterize the impact of the RHED grant(s) on your organization’s
overall technical capacity? Has administering your RHED grants increased your
organization’s ability to implement certain types of projects?
________________________________________________________________________
________________________________________________________________________
4. How would you characterize the impact of the RHED grant on your organization’s
overall administrative capacity? Has it improved your organization’s ability to handle
federal grants as well as other sources of funding?
________________________________________________________________________
________________________________________________________________________
5. Did you in any way see the RHED grant as complementary to funding you were
receiving or could receive from other Federal agencies?
Yes

No

Section IV: Leveraging
1. In your original grant application, you indicated (retrieve from application) as a source(s)
for obtaining leveraged fund support for your RHED project(s). Is this the funding source
you actually used at project start-up?
Yes

No

If no, why were you were unable to access those planned-for funds?
________________________________________________________________________
________________________________________________________________________
2. At project start-up, were you able to put in place any additional leveraged funds other
than those mentioned in your grant application?
Yes

No

If yes, please specify the additional source(s) and amounts of funding.
________________________________________________________________________
________________________________________________________________________
Also, if yes, did this additional funding allow you to expand or qualitatively enhance your
project design?
Yes

No

If yes, what changes did you make in your project design?
Page 8 of 11 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

________________________________________________________________________
________________________________________________________________________
3. Did winning any RHED grant enable you to access funding sources that you otherwise
may not have been able to access?
Yes

No

If yes, please specify the new sources, and explain how successful you have been in
obtaining funds from one or more of these sources.
________________________________________________________________________
________________________________________________________________________
4. How would you characterize the impact of your RHED grant(s) on your ability to
leverage funds for ongoing or future projects?
a. Has significantly increased the range of funding options open to us.
b. Has somewhat increased the range of funding options open to us.
c. Has had minimal impact on our ability to obtain additional funding.

Section V: Partnering-Linkages
1. How many formal partnerships (characterized by exchange of resources and/or signed
agreement) were created primarily because of your RHED grant(s)?
________________________________________________________________________
2. How important was securing an RHED grant in establishing these partnerships?
a.
b.
c.
d.

Could not have had relationship without RHED.
RHED was very important in establishing relationship.
RHED was a factor in establishing relationship.
RHED was not important in establishing relationship.

3. Will your organization be able to build upon the linkages established on the RHED
project for further collaboration?
a. Already involved in further collaboration.
b. Further collaboration has been planned but not implemented.
c. There has been no further collaboration since our RHED grant was completed.
4. Overall, how would you characterize the impact of your RHED grant(s) in enabling you
to establish productive working relationships with other groups in your community? Has
it helped you establish new partnerships with other local organizations?
________________________________________________________________________
________________________________________________________________________

Page 9 of 11 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

Section VI: Program Outcomes and Impact
1. What individuals or groups did you originally intend to target with your RHED
project(s)? What outcomes did you expect for these recipients?
________________________________________________________________________
________________________________________________________________________
2. How would you characterize your progress toward achieving these outcomes for
recipients?
________________________________________________________________________
________________________________________________________________________
3. Did your organization use any indicators to measure the progress of your RHED-funded
projects toward completion?
Yes

No

If yes, what data indicators have you used (are you using) to measure the progress of your
RHED-funded projects toward completion?
________________________________________________________________________
________________________________________________________________________
4. Currently, how would you characterize the capacity of your organization to collect and
analyze data for planning and management purposes?
a.
b.
c.
d.

High level of technical capability.
Adequate level of technical capability.
Marginal level of technical capability.
Less than adequate technical capability.

If you have a “high” or “adequate” level of capability, how important were your RHED
grant(s) to improving your capabilities in this area?
a. Extremely important.
b. Somewhat important.
c. Not important.
5. How would you characterize the overall impact of your RHED project(s) so far? How has
it impacted the community as a whole?
________________________________________________________________________
________________________________________________________________________
6. When you received your first RHED grant, did your organization have a strategic plan or
other statement of organizational goals?
Yes

No

7. If yes, how did you integrate your RHED project(s) into this plan? How did your RHED
project(s) support your organization’s overall goals?
Page 10 of 11 Pages
Econometrica, Inc.

January 5, 2014

Evaluation of the Rural Innovation Fund

1046-000/ DU205NC-13-T-0007

________________________________________________________________________
________________________________________________________________________

Section VII: Conclusion
Overall, what has your impression been of the RHED program? Do you have anything to
add about your experience that we have not covered here?
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

Page 11 of 11 Pages
Econometrica, Inc.

January 5, 2014