CDFI Fund, U.S. Department of the Treasury
Paperwork Reduction Act Submission
Revised October 7, 2010
TABLE OF CONTENTS: PART B AND APPENDICES
B1. Respondent Universe, Sample Selection and Expected Response Rates 1
B2. Procedures for the Collection of Information 6
B3. Methods to Maximize Response Rates and to Deal with Issues of Non-response 8
B4. Pre-testing of Procedures and Methods 10
B5. Individuals or Contractors Responsible for Statistical Aspects of the Design 11
List of Appendices
1. Guides for case Style Data collection
CDE Discussion Guide
CDE Information Sheet
QALICB Discussion Guide
QALICB Information Sheet
Investor Discussion Gide
Local Stakeholder Discussion Guide
2. QALICB Survey
3. Community/Economic Development Officials Survey
4. Federal Register Notice
5. Introductory Letters
Sample introductory letter from CDFI Fund to CDEs for case-style interviews
Sample letter from Urban Institute to CDEs for case-style interviews
Sample introductory letter from CDFI Fund to QALICBs for survey
Sample Urban Institute letter to QALICBs for survey
Sample introductory letter from CDFI Fund to local CD/ED officials for survey
Sample Urban Institute letter to CD/ED officials for survey
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
Three separate data collection efforts employing statistical methods are being conducted. The respondent universe for each is as follows:
For the case-style semi-structured telephone interviews, projects are the units of analysis. The respondent universe consists of key participants and stakeholders (see below) who are associated with all projects involving NMTCs that were allocated in the first four allocation rounds that had reported at least one financial transaction prior to the end of 2007. This is based on the CDFI Fund’s CIIS file as of December 2008.
For the online QALICB survey, the respondent universe consists of representatives of all QALICBs associated with NMTC projects that were allocated in the first four allocation rounds that had reported at least one financial transaction through the end of 2007, minus representatives of those QALICBs associated with projects that are randomly sampled and selected for the case-style telephone survey, above.
For the online local community/economic development officials survey, communities are the units of analysis. The respondent universe consists of one community and/or economic development official per community in all communities (i.e., cities or counties) in which at least one NMTC project had been initiated using tax credits allocated in the first four rounds that had reported at least one financial transaction through the end of 2007. There are 617 unique communities that meet these criteria.
Samples will be selected for each of the three surveys indicated above.
Case-style semi-structured telephone interviews. A sample will be drawn from the project universe identified above that will yield completed interviews with key participants and/or stakeholders of 80 projects. For each project, an average of four participants and/or stakeholders will be identified for interview purposes. The object of the interviews is to learn about the NMTC program’s role in attracting capital to economically distressed communities, how that capital was used (i.e., in what kinds of projects), project outputs and outcomes, and the extent to which NMTCs were needed for those outputs and outcomes to occur.
For budgetary reasons, the number of projects that can be studied through intensive case-style interviews is limited to 80. As will be discussed below, this sample size is large enough for estimating to the full universe of NMTC projects with a reasonable degree of confidence. The sample size may not, however, produce equally reliable estimates for certain sub-group analyses.
The sampling procedure consists of dividing the universe of projects into three strata—defined by the CDFI allocation round from which each project investment was made:
Round 1 (allocated in 2003);
Round 2 (allocated in 2004); and
Rounds 3 through 4 (allocated in 2005 and 2006).
Projects will be randomly selected within each stratum, with each project having an equal probability of selection; however, the number of projects selected from each stratum will not be in proportion to the distribution in the universe of projects. A smaller-than-proportionate number of projects will be selected from Rounds 1 and 2 and a larger-than-proportionate number will be selected from Rounds 3 and 4—as shown hypothetically in Table 3, below. In that example, 19 percent of the sample might be from Round 1 as compared to 39 percent of the universe, 19 percent might be from Round 2 as compared to 33 percent of the universe, and 62 percent might be from Rounds 3-4 as compared to 28 percent of the universe. If this were the case, each Round 1 and Round 2 project would have a 2 percent chance of selection into the sample while each Round 3-4 project would have a 9 percent probability of selection.
The rationale for disproportionate sampling by allocation round is that Round 1 and Round 2 projects are considered by both officials of the CDFI Fund and a number of external stakeholders to be somewhat atypical of how the program has evolved over time and, therefore, unrepresentative of the likely future direction of the program. In part, this is because some early round CDEs invested in projects that were already in the pipeline when the NMTC program began and because both Round 1 and Round 2 CDEs, according to the CDFI Fund, entered into looser allocation agreements than CDEs in later rounds (e.g., there were fewer requirements to invest in highly distressed communities and no prohibitions against refinancing of real estate). In the later rounds, agreements were tightened. The aim, then, is to select a sample that emphasizes the program as it has been evolving for some analytic purposes but that also allows for generalization to the full program history for other purposes.
It is important that the sample consist of projects that are far enough along in their evolution to be able to provide completed transaction details and evidence of (at least) intermediate outcomes. That is, they should be completed by the time interviews are conducted so that outcomes associated with them can be evaluated empirically as opposed to prospectively (using projected information). Although the CIIS does not contain information indicating if projects are completed, the data in Table 3 show that most projects that would be sampled originated in 2006 or earlier. Since interviews with project stakeholders will not occur until 2010, it is reasonable to presume that they will have been completed by then.
For purposes of extrapolating to the full universe of projects, the sample will have to be properly weighted to account for the disproportionate selection across strata. The total (weighted) sample will permit generalization to the program as a whole while. At the same time, because a larger-than-proportionate share of the projects will be from later rounds, there will be a sufficient number of later-round projects to permit reliable generalization to the program as it has been evolving over time.1
Table 3: Distribution of Year of Project Originations, Projects in the Universe and (Hypothetical) Projects in the Sample, by Allocation Round*
Allocation Round |
Year of Project Originations** |
Projects in the Universe |
(Hypothetical) Projects in the Sample |
||||||
2003 |
2004 |
2005 |
2006 |
2007 |
Number |
Percent |
Number |
Percent |
|
1 |
31 |
213 |
219 |
215 |
113 |
791 |
39% |
15 |
19% |
2 |
1 |
49 |
277 |
209 |
126 |
662 |
33 |
15 |
19 |
3 – 4 |
0 |
1 |
14 |
188 |
374 |
577 |
28 |
50 |
62 |
Total |
32 |
263 |
510 |
612 |
613 |
2,030 |
100% |
80 |
100% |
* This is based on CIIS files available to the Urban Institute as of December 2008.
** A NMTC project may consist of multiple transactions spanning more than one year. The year of project origination recorded here is the year of the origination of the first transaction associated with a project.
Because there are three other important project-related factors that need to be accounted for, the universe of projects will be further stratified by location (rural vs. urban), business purpose (real estate, business and mixed purpose), and CDE Parent Type (non-profit vs. for-profit) and, then, sampled randomly in proportion to their distribution in the universe. This will ensure the appropriate proportions of each such category in the sample.
Online QALICB survey. From the QALICB universe identified above, a random, equal probability sample of 380 QALICBs will be selected. However, appropriate compensation will be made for the fact that the universe consists of QALICBs remaining after the sample has been drawn for the case-style telephone survey—the latter consisting of a disproportionately under-sampling of early round projects and a disproportionately over-sampling of later round projects. This means that the remainder, from which the QALICB sample will be drawn, will contain slightly more early round projects and slightly fewer later round projects than their proportions in the QALICB universe.
For analytic purposes, the sample size of 380 is sufficient to ensure inclusion of all major types of QALICBs, from all four allocation rounds, and associated with all types of NMTC projects.
Online local community/economic development officials survey. From the universe of 617 communities identified above, a random, equal probability sample of 380 communities will be selected. These will consist of both urban and rural places.
Prior to administration of the survey, a separate list will be prepared of community or economic development agencies/offices/officials known to have an interest in, or responsibility for, those places. In some instances, such agencies/offices/officials will have responsibility for state-wide or region-wide coverage and, in others, county-wide or city/town-wide coverage. And, in some instances, there will be multiple agencies/offices/officials having responsibility over the same places. Where there are multiple such agencies/offices/officials, a case-by-case determination will be made as to which agency/office/official is most appropriate for purposes of responding to the survey. One agency/office/official will be surveyed per sampled community.
High rates of response are expected for the three data collection efforts. This is based in part on the fact that prospective respondents have an interest in either the NMTC program or in the use of Federal government programs for promoting local community and economic development interests. It is also based on the CDFI Fund’s strong support for the program evaluation, of which the data collection efforts are an integral part. It is expected that the CDFI Fund will communicate generic encouragement of participation to CDEs and others through various media, which is likely to have a positive effect on response rates. Note, however, that the CDFI Fund will not know which entities have been selected to be sampled in any of the data collection efforts and, therefore, will not take any action to encourage individual CDEs, investors, QALICBs, community/economic development officials, or other stakeholders to participate.
Case-style semi-structured telephone interviews. The cooperation of the vast majority of persons associated with the sampled projects is anticipated—as was the case with a pilot review of five NMTC projects completed by the Urban Institute in June 2007.2 Recently completed pretesting did not include communication or encouragement by the CDFI Fund, yet CDEs were willing to participate. QALICBs and other stakeholders were in some instances more difficult to reach but were generally receptive once contacted. Overall, full or nearly full participation is expected.
Online surveys. For the QALICB and local community/economic development officials’ surveys, a 75 percent response rate is expected. This estimate is based on previous, similar survey research completed by Silber & Associates and the Urban Institute and on the efforts that will be taken to encourage participation—as discussed below in Section B2. Therefore, the sample of 380 QALICBs and 380 local community/economic development officials is anticipated to produce 285 completed responses per survey.
Reliability of estimates. Both qualitative and quantitative analysis will be undertaken for the evaluation.
Case-style telephone interviews. The case-style telephone interviews will primarily provide qualitative information. However, where quantitative analysis can be done based on the interview data, the sample size will produce estimates at a reasonable level of precision—given the objectives of the evaluation. Data reliability depends, in part, on sampling error, which is a function of the statistical characteristics of any estimator, the number of cases on which the estimate is based, and sampling design effects resulting from stratification or weighting. Absent actual results, sampling errors for estimates to be derived cannot be calculated. However, presuming an equal probability sample design and a sample size of 80 for the case-style, semi-structured telephone interviews, a generalized technique was employed to calculate sampling errors for percentage estimates that may be near 5 (or 95) percent, 25 (or 75) percent, or 50 percent—at both the 95 percent and 90 percent confidence levels. At the 95 percent level, confidence intervals for an estimate of 5 percent can range from 1.6 percent to 13 percent; an estimate of 25 percent can range from 15.5 percent to 34.5 percent; and an estimate of 50 percent can range from 39 percent to 61 percent. At the 90 percent level, confidence intervals for an estimate of 5 percent can range from 1.9 percent to 11.5 percent; an estimate of 25 percent can range from 17 percent to 33 percent; and an estimate of 50 percent can range from 40.8 percent to 59.2 percent. For smaller sub-samples, of course, the confidence intervals will be larger, meaning that sub-group analysis will be limited to what is appropriate given the size of the sub-samples.
Online surveys. The online surveys are designed primarily to permit quantitative analysis. Assuming an equal probability sample design and 285 respondents for the online surveys, we calculates sampling errors for percentage estimates of 5 (or 95) percent, 25 (or 75) percent, or 50 percent—at both the 95 percent and 90 percent confidence levels. At the 95 percent level, confidence intervals for an estimate of 5 percent can range from 2.9 percent to 8.4 percent; an estimate of 25 percent can range from 20 percent to 30 percent; and an estimate of 50 percent can range from 44.2 percent to 55.8 percent. At the 90 percent level, confidence intervals for an estimate of 5 percent can range from 3.1 percent to 7.8 percent; an estimate of 25 percent can range from 20.8 percent to 29.2 percent; and an estimate of 50 percent can range from 45.1 percent to 54.9 percent. Similar to the case-style telephone interviews, confidence intervals for smaller sub-samples will be larger, meaning that sub-group analysis will be limited to what is appropriate given the size of the sub-samples.
In addition to sampling error, research results may be subject to other types of error, including issues of non-response or measurement relating to question wording or respondents’ ability to recall factual information or articulate answers. Note that there is no standard or statistical means to measure the consequences of these types of effects.
The plans for each data collection component are discussed below.
Case-style semi-structured telephone interviews. An average of four key participants or stakeholders associated with a sample of 80 NMTC projects will be interviewed.3 Generally they will be representatives of: (1) CDEs, (2) QALICBs, (3) investor entities and (4) local stakeholders—the latter consisting of a range of persons that may include attorneys, accountants, community or economic development officials, or others with a stake in, or knowledge of, the sampled projects.
The first contact to be made for each project will be the CDE, in the form of a letter from the Urban Institute, accompanied by a letter of endorsement of the program evaluation from the CDFI Fund. This will be followed by a telephone contact to the CDE to: (1) confirm their willingness to participate in the study; (2) request completion of an information sheet via a link that will be provided to a secure server; (3) schedule a telephone interview time; and (4) obtain the names and contact information of other key project participants—especially the QALICB and investor(s). The remaining participants/stakeholders will then be contacted via telephone and e-mail to attempt to schedule an interview time and, in the case of QALICBs, to request completion of an information sheet via a link that will be provided to a secure server.
Semi-structured discussions will be held with the various participants using guides that contain key topic questions (both open and closed ended) and strategic probes. This provides needed flexibility and allows for “drilling down,” where appropriate, for both capturing details about projects that will vary widely and seeking the evidentiary basis for initial responses that are provided. That information, combined with information available from the CIIS and supplementary information requested about non-NMTC financing, will provide the basis for: describing project-related activities, outputs and outcomes; describing community level outcomes; and assessing the extent to which NMTCs were needed for the investments.
In contrast to standard case studies that provide detailed, stand-alone descriptions of individual cases, the case-style interviews will be used to gather a rich and textured body of information that will be analyzed across projects. The information provided will be compared and contrasted for purposes of categorization and analysis using standard descriptive and inferential statistical methods—as opposed to individual case-by-case reports.
Online surveys. Two surveys will be developed for distribution and completion online.
Online QALICB survey. This survey will consist of primarily closed-ended questions that will take approximately 30 minutes to answer. It will be administered online by Silber & Associates Inc. Respondents will be notified by mail about the survey prior to the survey invitation (an “alert” letter). They will be informed that the survey is a part of an evaluation of the NMTC program and that they were randomly selected to participate in the survey. This will be followed by a second personalized letter inviting participation and providing the survey URL to potential participants. In this letter, respondents will be told that the survey is voluntary and will not affect in any way their participation in the NMTC program.
Online local community/economic development officials survey. The survey will consist of primarily closed-ended questions that will take approximately 15 minutes to answer. It will be administered online by Silber & Associates Inc. Respondents will be notified by mail about the survey prior to the survey invitation (an “alert” letter) informing them that the survey is a part of an effort to understand the role of Federal government economic development programs in local community and economic development, and that they were randomly selected to participate. This will be followed by a second, personalized letter inviting participation and providing the survey URL. Respondents will be informed that the survey is voluntary. The invitation and survey instrument will not focus on any particular NMTC project within the community but deal, instead, with generic issues related to the NMTC program.
Following notification of approval from OMB, the survey questionnaires will be entered into survey software, which will be pre-tested internally and externally to ensure seamless navigation, without malfunctions. Technical support by phone will be available to individuals completing the surveys from 9:00am-4:30pm ET, Monday-Friday, while the survey is being fielded. Surveys completed electronically will be uploaded into a Microsoft Access database by Silber & Associates. Silber & Associates will carefully track the outcome of all communication attempts with members of the survey sample and the final disposition of the sample—i.e., the number of sample members who participated in the survey, who refused to participate, or who could not be reached despite multiple calling attempts, etc.
The following methods will be used to ensure attainment of target response rates.
Case-style semi-structured telephone interviews. Advance letters, including a letter signed by the Director of the CDFI Fund, will be sent by the Urban Institute to key participants in the sampled projects, followed up by telephone calls and/or e-mails by Urban Institute researchers. Based on previous experience in conducting similar interviews with those who are involved in NMTC projects, including pretests of the information collection protocols, CDEs are generally quite interested in participating and their participation is helpful in facilitating the participation of others (QALICBs, investors, local stakeholders). Urban Institute experience with the NMTC program suggests that willingness to participate is less an issue than is establishing a mutually agreeable time to be interviewed. For that reason, the interviews will be done over a five-month period, allowing potential respondents an adequate amount of advance time to schedule an interview.
Online surveys. Online procedures have been designed to ensure high response rates for the QALICB and local community/economic development officials surveys. Respondents will first receive a survey ‘alert’ letter by mail (US Postal Service) sent by the Urban Institute/Silber & Associates. That will be followed a personalized survey invitation that includes a letter of introduction signed by the Director of the CDFI Fund and a letter from the Urban Institute that will contain a URL link, login name, and password provided to complete the survey on-line (see the Appendix for a sample of these letters). Silber & Associates will send an e-mail reminder one week after distribution of the survey invitation to those who have not responded. This should result in an increase in returned surveys for about 4-5 days. A second reminder will be emailed to non-respondents one week later. Once the return rate has again significantly declined (usually within days), non-respondents will be e-mailed the link to the survey with a shorter cover letter and a hardcopy reminder postcard printed on neon cardstock that emphasizes the importance of the study. A third e-mail reminder will be sent within one week to those who still do not respond, followed by telephone reminders from Silber & Associates. Telephone reminders will be made at different times of the day and days of the week. The survey website is available 24 hours a day, seven days a week. A toll-free technical support phone number will be listed at the site and on the printed survey letter in case a respondent has questions about the research study or encounters difficulty navigating the survey of the Internet.
To test for the possibility of non-response bias, QALICB non-respondents will be compared to QALICB respondents on characteristics for which CIIS data are available, including the following: geographic location of the QALICB; total amount of NMTC financing; percent of total financing accounted for by NMTC investment; and date of project origination. Local community/economic development officials who are non-respondents will be compared to respondents with respect to community characteristics such as region of the country and size of jurisdiction. If differences beyond the standard error are observed, consideration will be given to applying a non-response weighting adjustment.
Survey instruments and case-style interview guides were pre-tested in October and November 2009. Samples for each data collection effort were drawn prior to the pretest and pre-test respondents were selected from among the group of non-sampled potential respondents.
Case-style semi-structured telephone interviews. Four NMTC projects were selected for pre-test purposes. Related to these projects, interviews were conducted with: four CDEs, three QALICBs, two investors, and two community stakeholders. Each of the four case-style discussion guides was pretested as well as (a) the CDE and QALICB information sheets and (b) procedures for contacting prospective respondents.
Interviews with CDEs ranged from 45 to 75 minutes. Interviews with QALICBs ranged from 30 to 60 minutes. With respect to the data requested on the information sheet, CDEs and QALICBs indicated either that the data requested (particularly financial statements) was readily available or that they did not have the information (e.g., certain outcome measures). The pretest experience indicated that, in some cases, the CDE was the best source of the financial and/or outcome data while, in other cases, the QALICB had the requested information. For this reason, both CDE and QALICB information forms are included in the information collection. Since CDEs will be contacted first, if they are able to provide the project financial statements the information will not be requested from the QALICBs. Interviews with lenders and other stakeholders averaged 30 minutes.
Based on the pretest, a number of refinements were made to the data-collection materials and procedures. In particular, instruments were shortened and rearranged to improve the flow of the discussions. The investor interview was shortened substantially based upon the pretest experience where it was learned that investors may have limited knowledge related to the nonfinancial outcomes of a project. Revisions to the information sheets included clarification of terminology and instructions to provide the latest financial statements for the project rather than completing a new form requesting financial information.
Online Surveys. Pretesting of survey instruments involved the following activities: (1) initial calls and e-mails soliciting participation in the pretest; (2) formatting the survey as a Microsoft Word form that could be emailed and completed on-line; (3) e-mailing the survey instrument and confirming a time for a debriefing call; and (4) conducting a debriefing call to obtain respondent input about the length, clarity of questions, and overall reaction to the survey (respondents were requested to return their completed surveys prior to the debriefing call). A total of 3 QALICBs and 4 community/economic development officials participated in the pretest.
QALICBs indicated that the survey took 25 to 30 minutes to complete. Changes made to the online survey of QALICBs based on the pretest included minor revisions/clarifications to the introductory text, financing questions, and real estate unit counts. Equity financing was added as an option to several questions at the suggestion of pretest respondents. Annual revenue questions were clarified, and questions on tenant employment levels were included, but with instructions indicating that estimates are acceptable.
Community/economic development officials said their survey took 10 to 15 minutes to complete. Changes to the community/economic development officials survey based on the pretest included a minor change to the introduction and clarification of some terminology. Space was added to some of the survey questions for open-ended comments/explanations (e.g., other, specify) at the suggestion of pretest participants.
The agency responsible for receiving and approving contract deliverables is:
The Community Development Financial Institutions Fund
U.S. Department of the Treasury
601 13th St, NW
Washington, DC 20005
Persons Responsible:
Dr. James Greer, (202) 622-8258, [email protected]
Dr. Greg Bischak, (202) 622-8669, [email protected]
The organization responsible for administering the online surveys with QALICBs and local community and economic development officials is:
Silber & Associates
P.O. Box 651
Clarksville, Maryland 21029-0651
Person Responsible:
Dr. Bohne Silber, Principal, (410) 531-2121, [email protected]
The organization responsible for statistical design of data that will be collected is:
The Urban Institute
2100 M Street, NW
Washington, DC 20037
Persons Responsible:
Mr. Rob Santos, Senior Methodologist, (202) 261-5904, [email protected]
Mr. Timothy Triplett, (202) 261-5579, [email protected]
The organization responsible for conducting telephone-based case-style interviews and analyzing all data to be collected is:
The Urban Institute
2100 M Street, NW
Washington, DC 20037
Persons Responsible:
Ms. Nancy Pindus, Co- Principal Investigator, (202) 261-5523, [email protected]
Dr. Martin Abravanel, Co-Principal Investigator, (202) 261-5834, [email protected]
Appendix 1:
Guides for Case Style, Semi-Structured Telephone Interviews
CDE Discussion Guide
CDE Information Sheet
QALICB Discussion Guide
QALICB Information Sheet
Investor Discussion Guide
Other Local Stakeholder Discussion Guide
1 Sub-group analyses may, of course, be somewhat less reliable than total group analysis given the smaller sample sizes associated with the former. See Section B1.3.
2 Martin Abravanel, Nancy Pindus and Brett Theodos, Analysis of Selected New Markets Tax Credit Projects, The Urban Institute, June 2007.
3 All interviews will be conducted by experienced Urban Institute researchers.
| File Type | application/msword |
| File Title | Table of Contents |
| Author | US Department of Treasury |
| Last Modified By | US Department of Treasury |
| File Modified | 2010-10-19 |
| File Created | 2010-10-19 |