SUPPORTING DOCUMENT A
Web Survey for the Section 108 Evaluation
The U.S. Department of Housing and Urban Development’s (HUD) Section 108 program is the loan guarantee provision of the Community Development Block Grant (CDBG) program. It provides an upfront source of community and economic development financing, allowing an entitlement grantee to borrow up to five times its annual approved CDBG entitlement amount. Grantees address housing, community development, and economic development needs of low- and moderate-income persons and communities. The Section 108 loan guarantee program facilitates the financing of physical and economic revitalization projects—such as neighborhood commercial centers, small business incubators, industrial park rehabilitation, affordable housing activities, or office center construction—that have the potential for renewing neighborhoods or providing affordable housing to low- and moderate-income persons. HUD acts as the guarantor of a Section 108 loan made from private-market funds, promising investors that the loan will be repaid. This funding mechanism is considered by HUD to be “one of the most important public investment tools that HUD offers to local governments.”
The results of the Section 108 grantees1 will allow HUD to better understand the actual project operations and outcomes. The following is to request the Office of Management and Budget’s (OMB) approval for a new information collection entitled “Web Survey of the Recipients of Section 108 Funds.”
1. Explain the circumstances that make the collection of information necessary.
This is a request for clearance of a Web survey to be conducted to complete an evaluation of the Section 108 program. HUD’s Office of Policy Development and Research (PD&R), which provides the agency with research that influences agency policy development, was asked to conduct the evaluation in response to OMB’s 2007 Program Assessment Rating Tool (PART) evaluation of the Section 108 program.
The evaluation contract was awarded by HUD to Econometrica, Inc., a small business located in Bethesda, Maryland, that has conducted evaluation research for the past 10 years for various Federal government agencies. Econometrica has a history of conducting surveys and analyzing survey results, including those of individuals, public housing agencies, financial institutions, and other private firms.
To promote improved federal management and increased efficiency and effectiveness of federal programs, the federal government enacted the Government Performance and Results Act of 1993 (GPRA) that required agencies to set goals and report annually on program performance. The U.S. Office of Management and Budget (OMB) developed the Program Assessment Rating Tool, (PART), to monitor compliance with GPRA and rate program effectiveness. Since FY 2003, OMB has rated the performance of over 1,000 federal government programs. In 2007, the PART evaluation for the Section 108 program raised concerns about whether the program is redundant or duplicative of other government programs—and whether the program’s accomplishments and outcomes can be more effectively measured and tracked. The PART evaluation found that the Section 108 program:
Lacks long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program.
Has not demonstrated adequate progress in achieving its long-term performance goals.
Provides insufficient evidence to draw a strong conclusion regarding how Section 108 compares to other similar programs.
Has an inherent weakness relative to better designed credit loan guarantee programs—the federal government bears 100 percent of any losses; private lenders do not share the risks of loss from default, suggesting that the program generally encourages private lenders to exercise less caution than they otherwise would.
The evaluation is designed to answer three core questions:
Does the Section 108 program overlap with economic development programs operated by other federal agencies?
The issues to be examined here are the (a) extent and (b) nature of the overlap between Section 108 and other federal programs and (c) whether such overlap is functional or not. In addition, we will do some brief analysis on whether subsidizing 100 percent of the loan has inherent weaknesses.
What types of projects are being funded through Section 108 guaranteed loans?
Different communities have used the program to support a wide range of projects such as neighborhood commercial centers, small business incubators, industrial park rehabilitations, affordable housing activities, and office center construction. The questions to be examined here are what types of projects are most commonly funded and what role does Section 108 have in these projects?
What are the results of the Section 108 projects?
What quantifiable and qualitative results have Section 108-backed investments produced—ranging from jobs created to local revenues generated, to sites cleaned and cleared, to buildings constructed, and infrastructure upgraded?
Four tools will be used to provide data on the Section 108 program from FY 2002 to FY 2007, including information on project outcomes. The integrated database and the site visit component are expected to complement and enhance the Web survey. The survey is one of the methods this evaluation is using to obtain information on program effectiveness and efficiency. Often surveys will provide the only source of reliable information. The three tools are:
The integrated database will capture the basic information, available from multiple HUD sources, on the Section 108 projects funded from FY 2002 through FY 2007. After a preliminary review of the administrative data and HUD data systems, we are aware that these will not be sufficiently complete to form the sole basis for an evaluation of the Section 108 program.
The Web survey cuts across all three of the overarching research questions. The Web survey will be used to supplement and correct the information available through the HUD Program Office (i.e., administrative files and electronic databases). It will provide a systematic source of both quantitative and qualitative data on the types of projects being funded under Section 108 and capture the project accomplishments, financing, and outcomes. To a lesser extent, the survey will provide information on grantees’ views of the efficiency and effectiveness of Section 108 and will explore the overlap issue.
The site visits will provide qualitative data that allow us to probe for more in-depth information than can be obtained through a survey. The site visits will help us to fully understand the nature and effectiveness of the Section 108 program in a small number of sites. Analysis of site visit data will help us to write a more insightful and comprehensive final report, and may lead to better recommendations for program operations and future research in this area. The two preliminary site visits have already helped us to understand the nature of information maintained locally on Section 108 outputs and outcomes. Additional quantitative data, if available, will be collected during the site visits. To date we have completed two preliminary site visits.
Special studies of atypical Section 108 projects of grantees are also planned. Based on what we have learned as a result of the administrative file examination, there are some Section 108 projects that will need to be treated differently during data collection and analysis.2 We estimate that approximately 40 atypical projects receiving Section 108 funds during the study period will require the use of modified data collection and analysis methods. There are two planned conference calls to Puerto Rico HUD Office to explain the special nature of the Section 108 projects in PR. A case study of "mega projects" in Pennsylvania is also planned.
HUD requires reliable, comprehensive information regarding program performance in order to comply with the Government Performance and Results Act of 1993, fulfill reporting requirements for Congress and other stakeholders, measure progress made in meeting Business and Operating Plan goals, monitor grantee performance, and evaluate overall program results. The CDBG program’s performance measurement system—the Integrated Disbursement and Information System (IDIS)—does not incorporate Section 108 into the system. This evaluation will help HUD take positive steps to incorporate the Section 108 program into IDIS.
This data collection activity is authorized under [12.U.S.C. 1701z-1], which reads as follows:
TITLE 12--BANKS AND BANKING
CHAPTER 13--NATIONAL HOUSING
Sec. 1701z-1. Research and demonstrations; authorization of
appropriations; continuing availability of funds
The Secretary of Housing and Urban Development is authorized and directed to undertake such programs of research, studies, testing, and demonstration relating to the mission and programs of the Department as
he determines to be necessary and appropriate.
The information sought through this survey supports the U.S. Department of Housing and Urban Development’s Strategic Goal to Promote Decent Affordable Housing and Strengthen Communities.
2. Explain how, by whom, how frequently, and for what purpose the information will be used. If the information collected will be disseminated to the public or used to support information that will be disseminated to the public, then explain how the collection complies with all applicable Information Quality Guidelines.
The information will be used by HUD in part to address the programmatic issues raised by OMB’s PART evaluation that relate to program design. In 2007, the PART evaluation for the Section 108 program raised concerns about whether the program is redundant or duplicative of other government programs—and whether the program’s accomplishments and outcomes can be more effectively measured and tracked. The PART evaluation found that the Section 108 program:
Lacks long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program.
Has not demonstrated adequate progress in achieving its long-term performance goals.
Provides insufficient evidence to draw a strong conclusion regarding how Section 108 compares to other similar programs.
Has an inherent weakness relative to better designed credit loan guarantee programs—the federal government bears 100 percent of any losses; private lenders do not share the risks of loss from default, suggesting that the program generally encourages private lenders to exercise less caution than they otherwise would.
The information will be presented in an Evaluation conducted by Econometrica, Inc., a small business firm that has conducted research for Federal government agencies for more than 10 years. The proposed Web-based survey is a one-time only survey. The recipients of the survey are CDBG grantees who are beneficiaries of the HUD funds, and as such are committed to reporting accomplishments to the Department.
The Department is requesting a one-time approval to implement this Web survey. The responses to questions are expected to explain the results of the expenditure of Section 108 funds by providing clear measures to demonstrate progress toward contributing to each of the Section 108 program’s three major goals: “Expanding Access to Affordable Housing,” “Fostering a Suitable Living Environment,” and “Expanding Economic Opportunities” for low- and moderate-income populations. The survey will also provide information to determine whether there has been any change since the grantee’s application in terms of beneficiaries, scope, or location of activities.
The data will be analyzed to provide a context for program policy considerations. The information will be used to address comments from OMB and respond to requests from Congress, but will also help HUD develop performance measures and implement procedures to collect accomplishment data from recipients of Section 108 funds.
The findings and recommendation resulting from this evaluation will be presented in a report to PD&R. PD&R typically disseminates studies and publications through HUDUSER. HUDUSER policies and procedures comply with all applicable Information Quality Guidelines.3
3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological techniques or other forms of information technology.
This data collection effort is a self-administered Web survey. Web-based surveys are now widely used in social science research of the type being proposed for this evaluation. The recipients have the ability to complete the survey using the Web at their convenience. The survey contains instructions, help screens, definitions and other aids that guide respondents through the process.
Section A of the Web survey contains pre-filled information which the grantee provided to HUD in its application. The survey will contain several features to help the respondents such as, radio buttons, drop-down lists, text boxes and a help feature that will allow responders to see further explanations and definitions of technical terms referenced in the survey.
Additional key features include:
Secure data entry.
User login save/review, and submit capabilities.
Navigation to access different sections.
Navigation to access different screens sequentially or non-sequentially.
Interactive features to address questions.
We will launch the survey via QuestionPro—an integrated system capable of sending personalized e-mail cover letters, tracking which of the respondents have not completed the survey, and sending out reminders.
4. Describe efforts to identify duplication.
No comparable data have been collected for the recipients of Section 108 program funds. Neither the information we are seeking to be able to assess program effectiveness nor any other similar information on the Section 108 grantees is available.
5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden.
Small businesses or other small entities are not part of the target population of the information collection. The information collection is limited to grantees receiving Section 108 funds from HUD, local government officials, and any entity receiving or administering Section 108 programs not the small businesses they are seeking to help.
6. Describe the consequences to the Federal program or policy activities if the collection is not conducted or is conducted less frequently.
This is a one-time Web survey. HUD does not have any other means to collect the information OMB is requesting. The information is deemed essential by OMB for HUD to assess the effectiveness of the Section 108 program.
7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.
requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;
in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
There are no special circumstances that would require this information collection to be conducted in any of the ways listed as part of this requirement.
8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.
In accordance with the Paperwork Reduction Act of 1995, the Department of Housing and Urban Development published a notice in the Federal Register in February 25, 2010 (Vol. 75, No. 37, pp. 8728-9) announcing the agency’s intention to request an OMB review of data collection activities for the Section 108 Program Assessment. A copy of the Notice is attached at the end of this document.
9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.
HUD does not provide remuneration to grantees.
10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.
Econometrica Inc. has established stringent procedures and safeguards for securing and protecting against inappropriate disclosure or release of confidential information that will be observed or collected during this study. The report that results from this study will not associate program outcomes with individual grantees. The data that Econometrica Inc. provides to HUD will be purged of information that would enable the Department to identify specific grantees.
The specific information gathered in this effort will be used only for the purposes of this research. All members Econometrica Team members that will have access to these data have formally signed "Assurances of Confidentiality" pledges.
The statutory authority related to HUD’s ability to conduct research through a contract with Econometrica, Inc. is summarized below:
1) Section 3(b) of the Department of Housing and Urban Development Act, as amended, 42 U.S.C. 3532, authorizes the Secretary to “conduct continuing comprehensive studies, and make available findings, with respect to the problems of housing and urban development.”
2)Section 7(r)(1) of the Department of Housing and Urban Development Act, as amended, 42 U.S.C. 3535, provides that appropriated funds “shall be available to the Secretary for evaluating and monitoring of all such programs . . . and collecting and maintaining data for such purposes.” Subsection (r)(2) of the act indicates that programs authorized under title I of the Housing and Community Development Act of 1974 are subject to the above provision. Subsection (r)(4)(a) of the act further provides that the Secretary “may provide for evaluation and monitoring under this subsection and collecting and maintaining data for such purposes directly or by grants, contracts, or interagency agreements.”
3) Section 502(g) of title V of the Housing and Urban Development Act of 1970, as amended, 12 USC 1701z-2 (g), authorizes the Secretary “to request and receive such information or data as he deems appropriate from private individuals and organizations, and from public agencies.” It further provides that “[a]ny such information or data shall be used only for the purposes for which it is supplied, and no publication shall be made by the Secretary whereby the information or data furnished by any particular person or establishment can be identified, except with the consent of such person or establishment.”
Some of the information essential for the survey is public information required by regulations governing the Section 108 program (24 CFR 570, Subpart M). Every CDBG grantee is required to establish and maintain at least three major categories of records: Administrative records that apply to the overall administration of the activities; financial records; and project/case files documenting the activities undertaken under the Section 108 program.
Each contact person selected to respond to the Web survey will be given a password to access and complete the survey. HUD’s Section 108 Program Director, the CPD Field Office Representative and Econometrica will provide a clear overview of the study’s purpose, reasons why it would be in the interest of the program to respond, an assurance as to the legitimacy of the survey, and provide the name of a person the respondent can contact if the respondent has questions. Typically, Econometrica follows these procedures when conducting research on behalf of its Federal government clients.
11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.
Most of the data required for the survey are required of any grantee (under 24 CFR 570, Subpart M) receiving Section 108 funds. However, at the present time HUD does not have a system in place for collecting the information in place at the present time. Some of the questions require the grantee to state opinions on program; these are not covered under 24 CFR 570. These are essential for answering the three overarching study questions which OMB is interested for this evaluation. The grantees’ opinions will be categorized and classified in the aggregate. HUD will not be able to identify any grantee. We will purge the data files we submit to HUD of all confidential information.
12. Provide an estimate in hours of the burden of the collection of information.
The estimates are based on the pre-tests conducted from January through June 2010. We pre-tested the Web survey with four grantees that received Section 108 funds, a HUD Economic Development Specialist, and 6 analysts on the evaluation team. Based on the pre-test, we estimate that each grantee will take approximately 45 minutes to complete the Web survey. We estimate there will be approximately 300 respondents. The total burden hours are estimated at 225 hours (300 respondents * 45 minutes).
13. Provide an estimate of the total annual cost burden to the respondents or record-keepers resulting from the collection (excluding the value of the burden hours in Question 12 above).
This collection will not require the purchase of any capital equipment and will not create any start-up costs.
Based on information from HUD’s Community Development Specialists, we estimate that the contact person in charge of answering the survey makes, on average, $60 per hour including overhead. The total annual cost burden to the respondents is estimated at $13,500 (300 respondents*$60.00 average hourly wage including benefits*225 burden hours).
14. Provide estimates of annualized cost to the Federal government.
There are no additional costs to the government beyond the funds budgeted in the contract with Econometrica, Inc. to conduct the evaluation, including the survey. There are no other marginal costs to the Federal government for this data collection.
15. Explain the reasons for any program changes or adjustments.
There are no program changes.
16. For collections whose results will be published, outline the plans for tabulation and publication.
The results of the evaluation will be reported in the Final Section 108 Evaluation Report, which is scheduled to be delivered to HUD on December 21, 2010. In the final report, Econometrica will provide summary aggregated data and the evaluation results will not contain any personal information.
We will use SAS for most of the manipulations and analyses, while Microsoft Excel and Access will be used for data collection. Relevant project statistics include:
Frequencies of Year, State, and Category.
Cross-tabulations for Year by State and Year by Category.
Sum and Average Section 108 Totals for Year, State, Category, Year by State, Year by Category, and State by Category.
Number of city-state combinations with only one Section 108 project.
Section 108 totals for all city-state combinations with more than one project.
“Leverage” – the amount and ratio of funds provided by other sources.
Proposed, revised, and actual objectives and goals.
The final report will be published by PD&R. PD&R typically disseminates studies and publications through HUDUSER. HUDUSER policies and procedures comply with all applicable Information Quality Guidelines. HUD Information Quality Guidelines were published November 18, 2002. See: http://www.hud.gov/offices/adm/grants/qualityinfo/fr4769n02-final.pdf.
17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.
HUD is not seeking this exemption.
18. Explain each exception to the certification statement identified in Item 19 of the
OMB 83-I.
There is no exception identified.
1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.
The respondent universe is composed of projects receiving funding awards under the Section 108 Loan Guarantee Program. The program has been in effect since 1978. A financial management control file provided by the HUD Section 108 Program Manager indicates a total of 1,571 projects that either now or at some time in the past were funded under the program.
For this evaluation, approximately 300 projects have been selected for analysis. The projects represent a census of all Section 108 projects, with a few exclusions already noted above, receiving funding awards during FY 2002 through FY 2007. The research design narrowed the evaluation period to these years because (i) it would be difficult to locate local staff with knowledge of projects implemented prior to FY 2002, (ii) and the projects implemented from FY 2002 through FY 2007 are recent enough to reflect current program activity, and (iii) the activities of Section 108 projects funded after FY 2007 would not be far enough along to provide usable data for this evaluation.
Time Period |
Number of Section 108 Projects Funded by HUD |
Total Funding Amount |
Prior to 2002 |
1,185 |
$5,166,845,346 |
From FY 2002 through FY 2007 |
300 |
$1,365,159,000 |
From FY 2008 to Present |
54 |
$ 396,717,000 |
2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
2.1 Sample Selection Method
We are sending the Web survey to virtually all Section 108 projects funded from FY 2002 through FY 2007, with the exception of some Section 108 projects that will need to be treated differently during data collection and analysis. They include:
Projects under investigation - We agreed to exclude projects under investigation at the request of the HUD IG. There are two such projects.
Wide ranging projects located in Puerto Rico - The projects are different in scope from other typical Section 108 projects. These are essentially composed of distinct Section 108 projects under an umbrella Section 108 funded project. Typically each sub-project is assigned to different grantee contact, depending on the location. Wide ranging projects can cover multiple localities, regions and even states. We propose to gather data and conduct in-depth discussions on project performance with the San Juan Field Office and the Atlanta Regional Office. These data, along with the administrative file review data, will be used to conduct a parallel analysis of Puerto Rico’s Section 108 activity for inclusion in the final report.
Commonwealth of Pennsylvania Consortium projects – These multifaceted projects that deviate from projects receiving Section 108 funds. This analysis will describe the structure of the Consortium, an analysis of the individual transaction, including a description of the projects funded the activities funded and the benefits derived.
2.2 Expected Response Rate
According to experts in the survey research field, it is difficult to predict the level of survey participation as rates can be influenced by many factors how the invitation is worded, perceived benefit from participating in survey, and demographics. Research on Web surveys provides many examples of web surveys that have achieved response rates of 50 percent or better. Researchers estimate that respondent rates can reach 85 percent when the respondent population is motivated.4
Typically HUD customers (e.g., CDBG grantees) tend to be responsive. They only need to know that HUD has placed a priority on the data collection, and that the information will be used. Techniques proposed for use in this Web survey, using two electronic reminders for non-respondents and a follow-up contact if necessary, are similar to techniques used in a HUD mail survey of Section 8 Housing Choice Voucher housing quality and satisfaction during 2000-2002. Response rates for that survey were approximately 50 percent over the three year survey period. This is a Web survey that pretest respondents thought easy to complete, we anticipate a similar or better response rate. Also those responding to this survey will be responding as part of their job and have a vested interest in the Section 108 Program, which should enhance the response rate. It is anticipated that at least 50 percent of the surveys will be completed by Section 108 respondents.
2.3 Previously Conducted Collection
There has been no previous data collection for the Section 108 program.
2.4 Procedure for Data Collection
Data on individual Section 108 projects will be collected through use of an Internet survey. The Internet service will be used to send the e-mail invitation, Web link to the survey, and a copy of the survey instrument for use locally. The Internet service will track response rates and provide statistics on responses while the survey is underway. The Internet service will provide a file that can be used for analysis at the conclusion of the survey.
For this survey, Econometrica will identify a “primary respondent” from each grantee who will receive the survey and be responsible for completing it. We identified the “primary respondents through a review of the administrative files at HUD. We will send the list to the HUD’s Section 108 Program Director who will ask the individual Field Office Directors for their assistance in: 1) providing a current list of these contacts for each of the Section 108 projects included in the survey, and 2) sending an email invitation to each of these contacts to participate in the survey.
Together with these materials we will send a set of procedures to be followed, and an Excel file identifying the separate Section 108 projects by Field Office. We will work with HUD staff to obtain the returned Excel files and to follow up on any missing or incomplete information.
The CPD Field Office Directors will be asked to provide the requested information for each Section 108 project (contact name, phone number, and email address) back to Headquarters within ten working days.
After OMB approval of the respondent burden package, we will directly notify the Section 108 Program Office Director in HQ that it is time to ask the CPD Field Office Directors to send pre-survey invitations to participate in the survey to the contacts, within five working days. Our notification will include suggested text for the invitation. Copies of our notification will be sent to the HUD’s Government Technical Representative (GTR) and the Section 108 Program Director (who will have already approved this process up-front). In order to keep response rates high, we will make absolutely necessary that the CPD Field Office invitations are sent within the prescribed five-day window. We also will ask that the Section 108 Program Director sign a separate Headquarters email invitation, which will serve as the formal launch of the survey. Additional key steps in the process include two e-mail reminders as needed to prompt a survey response, and if necessary a follow-up phone call from the Econometrica team to encourage response.
2.5 Estimation
Despite the measures described below to reduce non-response rates, it is likely that some sample units will remain unmeasured. We will measure response bias and use post-survey adjustment to reduce non-response error specifically targeted to Web surveys.
3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.
The accuracy and reliability of the collected data depend upon receiving a high rate of return from the respondents. To achieve a high response rate, four steps will be taken that are consistent with contemporary strategies in the field of survey research.
First, the survey distribution package is designed to be respondent friendly.
The survey is divided into sections in order to provide visual relief from a continuous listing of questions.
The questions are laid out in a format that is easy to read and that promotes answering.
The content of the questions is designed to ask for types of information that program recipients are required to maintain and should have readily available as required by 24 CFR 570.503.
The terms used in the survey (e.g., infrastructure, public facility) are widely known among program recipients and well defined in technical materials issued by HUD’s Office of Community Planning and Development.
Second, respondents are treated as real people with distinct identities. The distribution mailings are addressed to specific respondents, rather than to a generic “grantee.” In the same way, the mailings are signed by HUD’s national Program Manager for the Section 108 program. In addition, the Econometrica Team will assure that both the Field Office and Headquarters invitations to participate in the survey are sent in the form of a personalized e‑mail, sent to the local contact person.
Third, the cover letter provides a clear and direct explanation of the reasons for the data collection and urges resident participation. The cover letter also encourages response by explaining that the sampling process is random and responses will be kept confidential. Offering confidentiality will also contribute to the accuracy and truthfulness of the responses. At present, local jurisdictions have few reporting responsibilities under Section 108, and some of our respondents may be uncomfortable reporting outcomes, if they think HUD will use the information for monitoring purposes. This is especially true where there have been delays or other problems. We would not want to create a situation in which only the successful projects are being reported to us. Also, some of our local Section 108 contacts may be reluctant to report any information whatsoever without first clearing the information with the mayor or city manager. Offering confidentiality may help address this concern. Finally, some of our questions involve expressing opinions of performance by the jurisdiction or by HUD. These opinions should not be reported or otherwise made available in a form that would reveal the identity of the person making the comment.
Fourth, we have plans in place to address issues of non – response. As noted above we plan follow-up by email and by telephone call. The first two follow-up actions are reminder e-mail messages. The e-mail reminders will be e-mailed at two weeks from the initial survey mailing to those who have not responded. The third follow-up is a telephone call from an Econometrica staff member to the respondent to encourage the respondent’s participation in the survey. Following a response to the Web survey, we will send a “thank you” e-mail message to each respondent.
We have examined how to handle item and unit non-response bias. We will make every effort to reduce the number of non- respondents in order to reduce the bias. We can evaluate unit non-response bias in several ways:
We can compare initial and late respondents on several key measurements. Sample statisticians theorize that those that respond after several attempts have some similarities with non-respondents. Differences would be considered an estimate of non-response bias.
We can compare the respondents and non-respondents on several dimensions (e.g., type of project, loan amount, location, year of loan, etc.). This is possible because we have comparable data on for the pool of grantees from HUD administrative files and other HUD data sources. Again if the comparison results in clear differences, we conclude that these differences indicate that we may have non-response bias in our data.
We do not anticipate having item non-response bias as we have taken precautions to reduce item non- response. Web surveys have the potential to help reduce the item non-response. For example, the Web survey does not allow respondents to accidentally skip a question. Several techniques can be used to deal with item non-response such as imputation and direct estimation. In some cases we will have HUD administrative data to impute values.
When reporting the results of this survey, we will include sufficient caveat of the limitations of the survey’s methodological approach.
4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.
The online instrument has been tested numerous times by members of the team. Following implementation of changes online, the instrument will be tested at HUD next week. We also will conduct pretests of the web instrument with nine Section 108 grantees that are not part of our sampling frame. We pre-tested the WORD version of the survey instrument with two Section 108 grantees in the Seattle HUD region. We also discussed the WORD version of the survey instrument with HUD field staff in Philadelphia and Seattle, and with two Section 108 grantees during the preliminary site visits.
5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
HUD contracted with Econometrica, Inc. (prime contractor) to conduct the Evaluation of the Section 108 Program. Mr. Martin Abravanel of the Urban Institute is Co-Principal Investigator with Ms. Priscila Prunella of Econometrica who also serves as Project Director. The survey instrument and analysis plan were developed by the Econometrica Team under the direction of Mr. Robert Gray, of Econometrica, Inc. an expert in HUD data collection systems and processes, in consultation with nationally known experts in the field of community development programs, including Chris Walker (Director of Research and Assessment for the Local Initiatives Support Corporation), Marcia Tonkovich (ICF International), Roger Frankoff (Consultant), and John Nagoski (Consultant). The national Section 108 Program Director, PD&R staff, and the HUD Field Office CPD staff also played an important role in the development of the data collection and analysis plan. Also consulted were HUD staff members, including the Section 108 Program Director, Paul Webster; the GTR, Dr. Judson James; the Director of the Evaluation, Research and Monitoring Staff, Kevin Neary; the Economic Development Specialist in the Philadelphia Field Office, Andrea Edwards-Spence; and the Economic Development Specialist in the Seattle Field Office, Donna Batch.
The actual collection of Web survey data will be performed through QuestionPro, a Web service that specializes in conducting Internet surveys. The Econometrica Team, including Craig Davis, Robert Gray, Chris Holleyman, Priscila Prunella, and Alex Thackeray from Econometrica, Inc., and Marty Abravanel, Nancy Pindus, and Brett Theodos from the Urban Institute, will be involved in the data collection and analysis.
The tables below show the names, affiliation, and contact information for those involved in the statistical design and the survey research.
Individuals Consulted on Statistical Aspects of the Design |
|
|
Name |
Affiliation |
Contact Information |
Chris Walker |
Local Initiatives Support Corporation |
202-739-9291 |
Marcia Tonkovitch |
ICF International |
610-525-0857 |
Roger Frankoff |
Consultant |
410-849-2324 |
John Nagoski |
Consultant |
540-657-7766 |
Marty Abravanel* |
The Urban Institute |
202-833-7200 |
Brett Theodos* |
The Urban institute |
202-833-7200 |
Nancy Pindus* |
The Urban institute |
202-833-7200 |
Robert Gray* |
Econometrica, Inc. |
301-935-5256 |
Chris Holleyman* |
Econometrica, Inc. |
301-657-9883 |
Priscila Prunella* |
Econometrica, Inc. |
301-657-9883 |
Alex Thackeray* |
Econometrica, Inc. |
301-657-9883 |
Craig Davis* |
Econometrica, Inc. |
301-657-9883 |
*Will also be involved in the data collection and analysis
HUD Staff who Advised on the Survey Design |
|
|
Name |
HUD Staff |
Contact Information |
Judson James |
HUD GTR |
202-402-5707 |
Paul Webster |
Section 108 Program Director |
202-708-1871 |
Donna Batch |
HUD Seattle Field Office |
206-220-5374 |
Andrea Edwards-Spence |
HUD Philadelphia Field Office |
215-861-7658 |
1 Although “grantee” typically implies a (CDBG) grant recipient, “grantee” has been used in past HUD publications to indicate Section 108 recipients or users. In this document, the term “grantees” refers to “Section 108 recipients,” “Section users,” or “Section 108 borrowers.”
2 All Section 108 projects for which we have information from any of the HUD data sources will be included in the integrated database described in Chapter 4.
3 HUD Information Quality Guidelines were published November 18, 2002. See: http://www.hud.gov/offices/adm/grants/qualityinfo/fr4769n02-final.pdf.
4 See: Cobanoglu, Cihan, Bill Warde, and Patrick J. Moreo. 2001. "A Comparison of Mail, Fax, and Web Survey Methods". International Journal of Market Research 43:441–52; Cook, Colleen, Fred Heath, and Russel L. Thompson. 2000. "A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys". Educational and Psychological Measurement 60:821–26; Couper, Mick P. 2000. "Web Surveys: A Review of Issues and Approaches". Public Opinion Quarterly 64:464–94; Couper, Mick P., Michael W. Traugott, and Mark J. Lamias. 2001. "Web Survey Design and Administration". Public Opinion Quarterly 65:230–53; Dillman, Don A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: Wiley; Dillman, Don A., Jon R. Clark, and Michael A. Sinclair. 1995. "How Prenotice Letters, Stamped Return Envelopes, and Reminder Postcards Affect Mailback Response Rates for Census Questionnaires". Survey Methodology 21:1–7; Mehta, Raj, and Eugene Sivadas. 1995. "Comparing Response Rates and Response Content in Mail Versus Electronic Mail Surveys." Journal of the Market Research Society 37:429–39; Schaefer, David R., and Don A. Dillman. 1998. "Development of a Standard E-Mail Methodology." Public Opinion Quarterly 62:378–97; Sills, Stephan J., and Chunyan Song. 2002. "Innovations in Survey Research: An Application of Web Surveys." Social Science Computer Review 20:22–30; Smith, Christine B. 1997. "Casting the Net: Surveying an Internet Population." Journal of Communication Mediated by Computers 3 (1): available online at http://www.ascusc.org/jcmc/vol3/issue1/; Witter, Scott G., Ruth Kline-Robach, David L. Long, Jon Bartholic, and Fred Poston. 2001. "MSU-WATER: A New Way of Addressing Water Quality Challenges." Water Resource Update 119:47–59; Schonlau et al, Conducting Research Surveys via E-Mail and the Web, Rand, 2002; and Dillman et al, Internet Mail and Mixed Mode Surveys, the Tailor Designed Method, Wiley, 2009.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Judson L. James |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |