Supporting Statement for the Evaluation of Microloan Program Outcomes
(OMB Control Number: 3245-XXXX)
1. Circumstances Making the Collection of Information Necessary 4
2. Purpose and Use of the Information Collected 5
3. Use of Information Technology and Burden Reduction 7
4. Efforts to Identify Duplication and Use of Similar Information 8
5. Impacts on Small Businesses or Other Small Entities 8
6. Consequences of Collecting the Information Less Frequently 9
7. Special Circumstances Relating to the Guideline of 5 CFR 1320.5 9
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 10
9. Explanation of Any Payment or Gift to the Respondents 11
10. Assurance of Confidentiality Provided to Respondents 11
11. Justification for Sensitive Questions 11
12. Estimates of Hour Burden, Including Annualized Hourly Costs 12
13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers 14
14. Annualized Costs to Federal Government 14
15. Explanation for Program Changes or Adjustments 15
16. Plans for Tabulation and Publication and Project Time Schedule 15
17. Reason(s) Display of OMB Expiration Date is Inappropriate 16
18. Exceptions to Certification for Paperwork Reduction Act Submissions 16
PART B. Collections of Information Employing Statistical Methods 16
1. Universe and Sampling Respondent Selection 16
2. Describe the Procedures for the Collection of Information 18
3. Describe Methods to Maximize Response Rates and methods to Deal with Issues of Non-Response 18
4. Describe Any Tests of Procedures or Methods to be Undertaken 19
List of Appendices
Appendix A-1 Project Schedule
Appendix A-2 60-day Federal Register Notice of Collection
Appendix A-3 Paperwork Reduction Act Submission 83-I
Appendix B-1 Borrower Web Survey
Appendix B-2 Borrower Telephone Interview
Appendix B-3 Borrower Introduction Email
Appendix B-4 Borrower Informed Consent
Appendix B-5 Borrower Reminder Scripts
Appendix B-6 Borrower Reminder Recruitment Email
Appendix B-7 Borrower Thank You Email
Appendix C-1 Intermediary Web Survey
Appendix C-2 Intermediary Telephone Interview
Appendix C-3 Intermediary Introduction Email
Appendix C-4 Intermediary Informed Consent
Appendix C-5 Intermediary Reminder Scripts
Appendix C-6 Intermediary Reminder Recruitment Email
Appendix C-7 Intermediary Thank You Email
Appendix D-1 Power Analysis
Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The Microloan Program is administered by the U.S. Small Business Administration (SBA). The purpose of the Microloan Program is to assist women, low income, veteran, and minority entrepreneurs, and other small businesses in need of small amounts of financial assistance. Under the Microloan Program, the SBA makes direct loans to Intermediaries (lenders) that, in turn, use the proceeds to make microloans and provide training and technical assistance to eligible Micro-borrowers in their SBA approved geographic service areas. The program’s mission is to integrate micro-level financing with training and technical assistance for start-up, newly established, and growing small businesses. By offering financing and technical assistance support to these businesses, the Microloan Program aims to support job creation and retention for small businesses that would have difficulty securing funding from conventional channels.
Currently, the SBA collects some administrative data for the job creation/retention, revenue, and business tenure of the Microloan Program borrowers, as well as for the technical assistance provided to Microloan Program borrowers. However, there is no process in place to systematically collect and track participant business outcomes over time or to measure participant experiences with the technical assistance received. A comprehensive effort to collect the administrative, survey, and secondary data is required to determine how the Microloan Program operates and to what extent it reaches its intended outcomes to grow participant businesses.
Indicate how, by whom, how frequently, and for what purpose, the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
For What Purpose the Information Is to Be Used?
The objective of this evaluation is to examine the relationship of the Microloan Program activities of lending and technical assistance to program outcomes of business revenue, job creation, and business survival. The evaluation will describe the population of micro-borrowers, their businesses and experiences with the program, as well as intermediary lender characteristics, and training and technical assistance provided to micro-borrowers. The results of the analyses of administrative, secondary, and survey quantitative data will be integrated to address the research questions and to identify suggestions for program improvement in providing loans, training and technical assistance, and promoting business growth and survival outcomes. The qualitative findings will provide explanatory insights into the findings produced by the quantitative analysis. The results will provide an in-depth understanding of the program processes, outcomes, and contexts, and inform future policy and program implementation changes to improve program participation, operations, and outcomes.
The methodological design for the evaluation involves the pre-post program assessment of business performance outcomes for a borrower’s business, including 1) number of employees, 2) revenue, 3) formation of new businesses that did not operate before the loan, and 4) existing business survival. Additionally, the impact of Microloan Program lending and technical assistance activities on program outcomes is also examined. This will involve the multivariate analyses to determine the magnitude and significance of the relationship of the training and technical assistance to the business growth outcomes. The evaluation will focus on borrowers who received Microloans during Fiscal Years (FY) 2010-2019 period. The change in business performance and tenure outcomes will be examined from the baseline, pre-program participation period (i.e., at obtaining the loan) to the two follow-up years (2019 and 2020). The change in business outcomes from the baseline to 2019 will determine Microloan borrowers’ business growth. The change in business outcomes from 2019 to 2020 will gauge the effect of the COVID-19 pandemic-induced economic crisis on business performance and survival and to identify characteristics of resilient small businesses that are more likely to survive the current economic crisis. All results will be presented in aggregate form in the final report and made publicly available on the SBA Program Evaluation & Evidence Registry (PEER) web page.
What information will be collected?
This new data collection effort attempts to minimize the respondent burden by maximizing the use of administrative and secondary data sources. SBA administrative data will be used for the baseline levels of business performance outcomes (revenue and employment). However, SBA administrative data for business performance outcomes are unavailable during or after the loan repayment period. Therefore, publicly available secondary data sources, web survey and interview responses will be used. These to-be collected business performance and training and technical assistance data are necessary to determine the extent to which the program achieves its intended outcomes to grow participant businesses.
There are four elements to the data collection effort: a borrower web-survey, borrower qualitative telephonic interview, intermediary lender survey, and intermediary lender telephonic qualitative interview (Appendices B-1, B-2, C-1, and C-2). The sample size for each data collection effort is detailed in Section 12 below. The data will also be used to measure participant experiences with the program’s lending, training and technical assistance, as well as their satisfaction with the program and suggestions for the program improvements. The survey data collection will be conducted by Optimal one time.
The survey and interview data collected will be used to address the research questions.
What are the job creation/retention, revenue growth and business tenure outcomes of Microloan Program borrowers?
Micro-borrower web survey will be used to measure these business outcomes for the borrowers with missing administrative or secondary data.
How do job creation/retention, revenue growth, and business tenure outcomes vary by business characteristic (such as business type, industry, geographic region, and underrepresented population)?
Micro-borrower web survey will be used to measure business characteristic (business certificates or designations) for the borrowers with missing administrative or secondary data.
What types, proficiency level, frequency, duration and delivery modes of technical assistance are being provided to Microloan Program borrowers?
Micro-borrower and intermediary web surveys will be used to measure characteristic of technical assistance being provided to and received by Microloan Program borrowers.
How does the type, proficiency level, frequency, duration, and delivery mode of technical assistance impact Microloan borrower job creation, revenue growth and business tenure outcomes?
Micro-borrower and intermediary web surveys will be used to measure perceived value and effectiveness of the program for Microloan program borrowers, satisfaction with the program, and suggestions for the program improvements.
Micro-borrower and intermediary telephone interviews will be used to determine factors affecting the effectiveness of training and technical assistance and provide suggestions for the program improvements.
From whom will the information be collected?
The data for this study will be collected from the total sample of 1,006 respondents, specifically:
A nationally representative sample of 820 borrowers who participated in the program FY 2010-2019 will be selected to participate in a self-administered web survey.
A random sample of 24 borrowers who participated in the program FY 2010-2019 will be selected to participate in the semi-structured telephone interview data collection.
Based on the assumed 80% response rate from the universe of 202 current and former intermediaries (see the study burden estimate, Exhibit 1 in section 12), 162 current and former intermediaries that participated in the program FY 2010-2019 are expected to complete a self-administered web survey.
Twenty-four current and former intermediaries completing the web survey will also participants in the semi-structured telephone interviews.
How will the information be collected?
The SBA has hired an independent contractor, Optimal Solutions Group, LLC (Optimal), to conduct the evaluation of the Microloan Program The data collection will involve web-based surveys and semi-structured telephone interviews. The data collectors for the telephone interviews will take detailed notes to document the responses. The surveys and interview scripts are presented in Appendixes B-1, B-2, C-1, and C-2. Email reminders will be sent to non-responding borrowers and intermediaries (Appendixes B-6 and C-6). Some non-respondents will be contacted by telephone to remind them to participate (Appendixes B-5 and C-5).
The data collection will begin one week after the OMB approval. For the intermediaries, the data collection will be conducted during the first month of the data collection period and include the following (see Exhibit 1 in section 12).
Email introduction from the SBA (Appendix C-3).
Three email follow-ups during the first week (Appendix C-6).
Three phone calls leaving answering machine messages during the second week (Appendix C-5).
Two email follow-ups during the fourth week (Appendix C-6).
For the borrowers, three waves of data collection are expected over three months data collection period, beginning one week after the OMB approval. To maintain the integrity of the sampling design and minimize biasing the sample, Optimal will conduct up to 9 contact attempts with non-respondents via both email and phone follow-ups before they are replaced with sampled borrowers from the same sampling stratum (see Exhibit 3). The use of 9 contact attempts before the non-respondent is replaced and the three waves of data collection will minimize the sampling bias; otherwise, the sample will be systematically biased toward those easily contacted and recruited. For the borrowers, the data collection will include the following (see Exhibit 1 in section 12).
Email introduction from the intermediary (Appendix B-3).
Three email follow-ups during the first week (Appendix B-6).
Three phone calls leaving answering machine messages during the second week (Appendix B-5).
Two email follow-ups during the fourth week (Appendix B-6).
How frequently will the information be collected?
The data collection will be conducted once.
Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden. Insert any applicable electronic web address.
The SBA is committed to complying with the E-Government Act of 2002 to promote the use of technology in order to lessen the burden of data collection. Survey data will be collected via web-based surveys, which facilitate the efficient collection and analysis of data. The web surveys will be self-administered using the web survey link individualized for each respondent. Web-based surveys reduce the burden placed on respondents because they ensure that respondents do not need to read through instructions to skip questions that are not relevant to them, as the skip pattern is automated. Furthermore, the web surveys allow the respondents to skip any items they are unwilling to answer. The web-based surveys also allow respondents to stop at any time and resume the survey at the point at which they last stopped. The questions for the business employment and revenue, as well as business certificates or designations will not be asked on the web surveys if they are available in the SBA data files or secondary data sources. The web-based survey requires no more than hitting a submit button to transfer the information to the database.
The respondent burden will also be reduced by minimizing the use of the open-ended items for web surveys. Detailed descriptions of intermediaries and borrowers’ experiences with the program will be collected via semi-structured telephone interviews. These interviews will gather in-depth contextual information that cannot be captured in an online survey and require an interaction with the interviewer telephonically to maximize convenience and to maintain social distancing.
Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.
The survey and interview data requirements for the evaluation have been carefully reviewed to determine whether the needed information is already available in the SBA administrative data or the secondary data sources. The SBA collects some administrative data for the job creation and retention, revenue, and business tenure of the Microloan Program borrowers and training and technical assistance provided through the intermediary lenders. However, there is no systematic process to collect or track long-term participant business outcomes or the types and benefits of training and technical assistance received. Additionally, after reviewing secondary data sources including Census, the Infogroup Historical Business Data, and Dun and Bradstreet (D&B), it was concluded that no existing data sources are available to answer the study’s research questions. Therefore, a comprehensive effort to collect the administrative, survey, and secondary data is required to determine how the Microloan Program operates and to what extent it reaches its intended outcomes to grow participant businesses.
If the collection of information impacts small business or other small entities, describe any methods used to minimize burden.
This data collection will obtain information from small businesses participating in the Microloan program. Data will also be collected from intermediaries, many of whom are themselves considered small by size standards. While most or all respondents are small businesses, they are minimally impacted as the burden per respondent is low (Exhibit 1).
To minimize the respondent burden, efforts have been made to simplify and streamline the surveys. Surveys will be disseminated via an online platform, allowing small business owners to take the survey at a convenient time, exit the survey if necessary and resume from the point that they last stopped and only answer questions that pertain to their experiences. To the extent possible, surveys will omit questions if the information is available from administrative or secondary data sources. The surveys minimize the use of open-ended questions. Open-ended questions will be asked for a small subset of participants (i.e. 24 microloan borrowers and 24 intermediary lenders) through semi-structured telephone interviews.
Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
The data collection is part of an evaluation that will allow the SBA to describe the extent to which the program is achieving its intended outcomes and develop suggestions for the program improvements. The data collection will be conducted once. Without this study, the SBA will not be able to examine the Microloan Program operations, intermediaries’ and borrowers’ characteristics, or the relationship of lending and technical assistance to business growth outcomes. As a result, the SBA may not have enough information to make informed decisions on future changes in the program policies and procedures to improve the program implementation and outcomes.
Explain any special circumstances that would cause an information collection to be conducted in a manner:
requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary, trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
There are no special circumstances. The collection of information is conducted in a manner consistent with the guidelines in 5 CFR 1320.5.
If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.
A notice of this data collection was published in the Federal Register on September 25, 2020, at 85 FR 60508 (see Appendix A-2). All comments were due on or before November 20, 2020. No comments on this collection were received.
Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.
SBA consulted with Optimal to design the methodological approach for the administrative data collection, the survey and the semi-structured instruments. Optimal functions as an independent third party to design the methodological approach for data collection and analyses, develop and pretest the data collection instruments, collect data, analyze the results, and produce the final report. Optimal’s team is composed of researchers with experience in diverse disciplines, including psychology, sociology, labor economics, and program evaluation (see Part B Section 5 below for a list of key Optimal staff).
Eight intermediary lenders were consulted to pretest the intermediary instruments. Six microloan borrowers were consulted during the instrument pretesting process for the borrower instruments. The pretest participants provided additional information on aspects of the data collection process and instruments such as the frequency, the clarity of instructions, completeness of responses options, disclosure, and additional data elements to record. The pretest participants also responded positively to the overall nature and purpose of the data collection effort (see Part B Section 4 below).
Explain any decision to provide any payment or gift to respondents, other than re-enumeration of contractors or grantees.
There will be no monetary or non-monetary incentives or gifts to the respondents.
Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
Optimal, an independent contractor, will be collecting, compiling, and analyzing all data collected for the evaluation. The data will be collected and stored on Optimal’s secure servers. In addition, under its contract with SBA, Optimal has agreed to implement administrative, physical, and technical controls to protect personal and sensitive information and to utilize data management security protocols to ensure restricted access to data and data privacy maintained on its system and in its reports.,
Assurance of confidentiality, to the extent permitted, is provided by the Privacy Act of 1974, 5 U.S.C. 552a and the Freedom of Information Act, 5 US.C. 552, exception (b)(4), which allows SBA to deny Freedom of Information Act(FOIA) requests for information disclosures that could cause substantial competitive harm, and exception (b)6), which protects records that would constitute a clearly unwarranted invasion of personal privacy if disclosed. Contractors are required to adhere to the Privacy Act, which is stated in the contract with Optimal. Data will be reported only in a tabular form, with analysis cells large enough to prevent individual respondents’ identification. The information obtained from the interviews will be attributed to anonymous participants. The surveys, interviews, and data files will not include identifying information, and respondents will be tracked using random number identifiers. Identifying information will be secured by Optimal and will only be released to Optimal’s internal staff members who need it for survey data-collection operations and data management procedures. The introduction letters, consent form, and the surveys contain the following statement: “Your responses will be kept strictly confidential, and your name will never be used in any reports produced from this study. All survey responses will be reported as aggregated statistics in the form of averages, percentages, and frequency counts. All survey responses will be stored as de-identified information and these data will be provided to the SBA.” (see Appendixes B-3, B-4, B-5, C-3, C-4, and C-5).
Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
The data collection does not request information typically considered private and sensitive in nature. However, survey respondents will be asked about their businesses’ employment and revenues in 2019 and 2020. The objective of this evaluation is to examine the relationship of the Microloan Program activities of lending and technical assistance to program outcomes of business revenue, job creation, and business survival. The evaluation will rely, as much as possible, on administrative and secondary data sources for this information. However, these data on employment and revenue are not available for all sampled participants and therefore, the survey respondents will be asked about their businesses’ employment and revenues in 2019 and 2020. Although potentially sensitive in nature, questions regarding business performance are essential in determining whether the program is achieving its intended outcomes. The information will be used to report participants’ business growth over time, from the year of the microloan initiation to 2019 and 2020.
Web survey instruments also include questions about age, gender, and education in order to fully describe the population of Microloan program borrowers and to statistically control for the owner demographic characteristics in modeling the key predictors of business growth outcomes. Although potentially sensitive in nature, questions regarding demographic characteristics are essential for the valid results of statistical modeling of the relationship of the Microloan Program activities of lending and technical assistance to program outcomes of business revenue, job creation, and business survival.
Provide estimates of the hour burden of the collection of information. Indicate the affected public, number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.
The SBA estimates that a maximum of 2,286 hours is needed to conduct this study. The total annualized hour and costs burdens are presented below in Exhibit 1. The affected public and respondent types for this data collection are composed of small business owners who received microloans and the program intermediaries servicing the microloans FY 2010-2019. Pretesting was conducted for the surveys and interviews to derive the respondent burden estimates (see Part B section 4 below). Based on the pretest, the estimated average completion time of a borrower web survey is 20 minutes and an intermediary web survey is 15 minutes. Each respondent type (borrower and intermediary) will require 20 minutes to complete the qualitative interview. In addition, research staff pretested all data collection forms and materials to provide a burden estimate for each type of respondent contact. Reading data collection materials was estimated at 1 to 3 minutes, depending on the material, as shown in Exhibit 1 below.
Based on a power analysis, the expected sample size for the borrower web survey was estimated to be 820 (see Section B below). Furthermore, based on the similar surveys conducted by Optimal for the SBA and the pretesting conducted for this evaluation, the estimated response rate is 10% for the borrowers. Since the total of 820 completed web surveys are needed, 8,200 borrowers will need to be invited to participate in the study (Exhibit 1). These drawn cases will also be used to recruit borrowers to complete the 24 phone interviews. Based on the pretesting of the data collection process and previous evaluation conducted by Optimal for the SBA, most of the small business respondents did not complete web surveys right away and initially had partially completed surveys. Therefore, the reminder phone calls will prioritize cases with partial completed web surveys and up to 10 percent of borrowers with emailed survey invitations may require phone call reminders. The reminder phone calls will also be used to recruit borrowers from strata with the lowest response rates (see Exhibit 3 in section B).
For the intermediaries, a full census of the 202 lenders will be conducted, and therefore, no sampling was required for this respondent population. Based on the pretesting, it was assumed that the 80% response rate will be achieved for 202 intermediaries, resulting in 162 completes, which includes 24 intermediaries who will complete both a web survey and phone interview. Based on pretesting, some intermediaries were not in the position to respond to the survey about the Microloan program, which required obtaining an alternative point of contact in the organization. Therefore, the reminder phone calls will attempt to obtain an alternative point of contact for some of the intermediaries with emailed survey invitations. The reminder phone calls will also be conducted for intermediaries with partially completed web surveys. Overall, it was estimated that up to 15 percent of intermediaries with emailed survey invitations may require phone call reminders.
The SBA estimated that the data collection's maximum cost to be $74,730 (Exhibit 1). The cost burden estimate for borrowers was based on an average annual income of $64,647 for small business owners or an hourly rate of $31.08.1 The cost burden estimate for lenders was based on 2019 National Occupational Employment and Wage Estimates United States, U.S. Department of Labor, Bureau of Labor Statistics, 11-1021 General and Operations Managers, mean hourly wage $59.15.2
Exhibit 1. Study burden estimate by respondent and instrument type
Type of data collection |
Number of respondents |
Frequency of data collection |
Average minutes spent per response |
Average burden per respondent (minutes) |
Total Annual Burden (hours) |
Total Annual Burden (dollars) |
Intermediary (lender) |
|
|
|
|
|
|
Email introduction from the SBA |
202 |
1 |
3 |
3 |
10 |
$597 |
Request to send email invitations to borrowers |
202 |
1 |
10 |
10 |
34 |
$1,991 |
Reminder emails from Optimal with web survey link |
202 |
5 |
2 |
10 |
34 |
$1,991 |
Reminder telephone calls by Optimal |
30 |
3 |
2 |
6 |
3 |
$179 |
Web survey completion |
162 |
1 |
15 |
15 |
40 |
$2,390 |
Telephone interview completion |
24 |
1 |
20 |
20 |
8 |
$473 |
Thank you email for completed data collection |
162 |
1 |
1 |
1 |
3 |
$159 |
Intermediary total |
202 |
|
|
48 |
132 |
$7,782 |
Borrower |
|
|
|
|
|
|
Email introduction from the intermediary with web survey link |
8,200 |
1 |
3 |
3 |
410 |
$12,743 |
Reminder emails from Optimal with web survey link |
8,200 |
5 |
2 |
10 |
1367 |
$42,476 |
Reminder telephone calls by Optimal |
820 |
3 |
2 |
6 |
82 |
$2,549 |
Web survey completion |
820 |
1 |
20 |
20 |
273 |
$8,495 |
Telephone interview completion |
24 |
1 |
20 |
20 |
8 |
$249 |
Thank you email for completed data collection |
844 |
1 |
1 |
1 |
14 |
$437 |
Borrower total |
8,200 |
|
|
40 |
2,154 |
$66,948 |
TOTAL |
8,402 |
|
|
|
2,286 |
$74,730 |
Note 1: the average intermediary respondent burden of 48 minutes reflects 138 respondents who will complete web survey only for the total of burden of 45 minutes and 24 who will complete both web and phone interview for the total burden of 65 minutes.
Note 2: the average borrower respondent burden of 40 minutes reflects that all borrowers will complete either web survey or phone interview.
Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information.
There is no other total annual cost burden to respondents or record keepers.
Provide estimates of annualized cost to the federal government. Provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.
The federal government's annualized cost is $180,303 (see Exhibit 2), including the costs associated with the contractor data collection efforts and the assigned SBA employees' salaries. The cost estimates for SBA employees are based on the salary for a GS-14 and GS-15 federal employees. Starting salary for a GS-14 employee is $121,316 per year at Step 1 and for a GS-15 employee is $142,701 per year at Step 1 in 2020 for the locality pay area of Washington-Baltimore-Arlington, DC-MD-VA-WV-PA. The hourly base pay of a Step 1 GS-14 employee is $58.33 per hour and GS-15 employee is $68.61 per hour.3 The SBA anticipated that these persons will work 80 hours per year for 1 year. The data collection cost for conducting the study is included in the contract between the SBA and Optimal Solutions Group, LLC, under contract number 73351018A0038. The evaluation data collection cost is $170,148 from 05/01/2020 through 11/01/2021.
Exhibit 2. Annualized costs to the federal government
Estimates |
SBA COR |
SBA advisor |
Evaluation contractor |
Total federal costs |
Number of employees |
1 |
1 |
NA |
NA |
Hours anticipated |
80 |
80 |
NA |
NA |
Average hourly pay |
$58.33 |
$68.61 |
NA |
NA |
Total cost |
$4,666 |
$5,488.8 |
$170,148 |
$180,303 |
Explain the reasons for any program changes or adjustments reported in items 13 or 14 of OMB form 83-I.
This is a new data collection; there are no program changes or adjustments.
For collections of information whose results are planned to be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.
Data analysis will provide the results addressing the research questions described in section 2 above and will involve the following.
Descriptive analyses of characteristics of Microloan borrowers and their businesses.
Descriptive analyses of characteristics of intermediaries.
Descriptive analyses of the various dimensions of training and technical assistance (type, frequency, duration, mode, etc.) provided to and received by Microloan borrowers.
The univariate analyses comparing Microloan borrowers’ business growth outcomes from the baseline (i.e., the loan initiation) to 2019 and 2020.
The multivariate analyses to identify the key drivers of business growth outcomes including dimensions of training and technical assistance provided to and received by borrowers, controlling for owner, business, and intermediary characteristics.
This study will use both descriptive and inferential statistical techniques to analyze the data. The descriptive statistics will include frequency, percentages, averages, standard deviations, and ranges for the data elements. The results are expected to include tables for the descriptive statistics and graphical elements, such as frequency distribution graphs, box-and-whiskers plots, pie charts, and maps. The inferential statistics will include univariate techniques, such as the t-test and chi-square test, as well as multivariate techniques, such as regressions and mixed models. Ordinary Least Squares regressions will be used to measure the change in continuous business performance outcomes (the number of employees and revenue) between baseline and the 2019 follow-up year. For the dichotomous business tenure outcomes (new business formed that did not exist before the program and the business closures) using logistic regressions.
Both types of models will determine the significance of training and technical assistance in predicting business outcomes while controlling for borrowers and intermediary characteristics, business characteristics, loan characteristics and experiences, and any other factors significant by the models. The inferential statistics results will include the standardized and unstandardized coefficients for the predictors, as well associated p-value for statistical significance at the .05, .01, and .001 levels. Standard errors and 95% confidence intervals will also be provided for the estimates.
Analyses of the qualitative data collected from intermediaries’ and borrowers’ interviews will involve coding interviews for the major themes and subthemes. The analyses will use a grounded theory approach. The intent is to move beyond descriptions to generate a theory that uses the qualitative data to explain of the relationships among a set of emerging themes, circumstances, and situations. The results will be used to describe the dimensions of training and technical assistance that are perceived as more or less valuable and effective by borrowers and intermediaries. Other expected qualitative findings will include the description of how experiences with the microloan program compare with other business lending and training programs, best practices for lending and training, as well as issues and problems with the program implementation and achieving outcomes. Integrating qualitative results with micro-borrowers’ and intermediaries’ survey data analyses explain and contextualize the quantitative results. The collective results allow the development of program improvement recommendations.
The data will be collected on a one-time basis in 2021 from 04/13/2021 to 7/12/2021. The complete schedule for the project is attached in Appendix A-1 Project Schedule.
The final evaluation report will be completed in October 2021 and published shortly thereafter on www.sba.gov/evaluation. A standardized schedule for the project is attached in the Appendix A-1 Project Schedule. The report will contain an executive summary, background and purpose, findings and results, discussion of results, conclusions and recommendations, and technical appendices. The report is also expected to provide programmatic recommendations based on the evaluation’s findings to improve the operations and future evaluations of the Microloan program. The set of formal recommendations will be aimed at addressing the following objectives: What can be done to improve borrowers' ability to achieve business growth? What improvements could be made to facilitate business owners' access to financing, especially those from underserved populations? How can the efficacy of technical assistance be improved to promote business growth outcomes?
If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
SBA plans to display the OMB expiration date on all data collection instruments.
Explain each exception to the certification statement identified in item 19, “Certification Requirement for Paperwork Reduction Act” of OMB Form 83-I. If Agency is not requesting an exception, the standard statement should be used.
SBA is not requesting any exceptions to the certification.
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
A national survey of microloan borrowers and intermediary lenders will be conducted to examine program outcomes and the impact of technical assistance and training on business growth. The SBA Microloan Portfolio Electronic Reporting System datafile was used to construct the sampling frame that contains 45,146 loans that borrowers received during the evaluation timeframe, Fiscal Year 2010-2019 (Exhibit 3). The frame was cleaned and deduplicated to develop the final sampling universe of approximately 36,000 unique borrowers.
The evaluation will use a stratified, random sample of micro-borrowers to provide nationally representative results. The number of strata was minimized based on the most important business and owner characteristics provided in the SBA administrative data, including: 1) rural/urban, 2) underrepresented owner (yes/no), 3) geographic region (Northeast, Midwest, South, and West). The sample size of 820 results in 55 cases per 12 strata and 80 cases per two West strata; sufficient for national and regional representation (see Exhibit 3).4 Based on the similar surveys conducted by Optimal for the SBA and the pretesting conducted for this evaluation, the estimated response rate is 10%. The sample size for the borrowers’ study is 8200. We will invite all 8200 to participate in the web survey and we expect to get 820 respondents and a separate 24 will also complete the qualitative interview.
Two additional variables will be used in the poststratification: business tenure (start-up (<2 years) or existing business (2+ years)) and the time period since the borrowers’ initial loan (during or after the loan repayment). Based on the distribution for these variables, stratification by these variables is not required and the representativeness of the sample will be ensured via poststratification by these variables, if needed. Depending on the final sample, poststratification may also adjust for the other factors, such as business age and type.
A finite population correction will be applied to the stratification and weighting design. Typically, sample sizes of less than 10% of the population size should be drawn to ensure independence and produce correct variance calculations. Given that the size of the population of borrowers is about 36,000 and the total sample size drawn is 8,200, this is more than double the ceiling for inference conditions to be met. However, since it is expected that most of the 8,200 businesses will not respond and the finite population correction will be employed, the sampling design is appropriate for inference and will result in national and regional representation.
Exhibit 3: Sampling Universe, Expected Response Rate, and Borrower Sample by Strata
Region |
Urban/rural |
Underrepresented owner |
Frequency borrowers |
Drawn sample size |
Expected response rate |
The sample size of completed surveys |
Northeast |
Rural |
No |
600 |
550 |
10% |
55 |
Northeast |
Rural |
Yes |
686 |
550 |
10% |
55 |
Northeast |
Urban |
No |
2,821 |
550 |
10% |
55 |
Northeast |
Urban |
Yes |
8,861 |
550 |
10% |
55 |
Midwest |
Rural |
No |
991 |
550 |
10% |
55 |
Midwest |
Rural |
Yes |
1,446 |
550 |
10% |
55 |
Midwest |
Urban |
No |
2,456 |
550 |
10% |
55 |
Midwest |
Urban |
Yes |
4,311 |
550 |
10% |
55 |
South |
Rural |
No |
875 |
550 |
10% |
55 |
South |
Rural |
Yes |
1,038 |
550 |
10% |
55 |
South |
Urban |
No |
2,238 |
550 |
10% |
55 |
South |
Urban |
Yes |
4,286 |
550 |
10% |
55 |
West |
Rural |
No and Yes |
255 |
N/A |
N/A |
N/A |
West |
Urban and rural |
No |
1,746* |
800 |
10% |
80 |
West |
Urban and rural |
Yes |
3,761* |
800 |
10% |
80 |
TOTAL |
|
|
36,117 |
8,200 |
10% |
820 |
Note: West Region Rural strata were small and had to be combined with West Urban strata, for a total of 14 strata.
Note *: includes West rural cases.
A census of all 202 SBA-approved Microloan intermediaries, including current and former intermediaries, for Fiscal Year 2010-2019 will be used for the data collection. An 80% response rate is expected for this part of the data collection, resulting in the 162 respondents with completed surveys, of which 24 respondents are expected to complete both a web survey and phone interview (Exhibit 1).
Statistical methodology for stratification and sample selection, estimation procedure, degree of accuracy needed for the purpose described in the justification, unusual problems requiring specialized sampling procedures, and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
As previously discussed, the intermediary lender sampling is not required because the surveys and interviews will be administered to the universe of all 202 intermediaries currently participating or previously participated in the Microloan Program during the evaluation timeframe Fiscal Year 2010-2019.
Based on a power analysis, the expected sample size for the micro-borrower survey was estimated to be 820, based on 80 percent power, an alpha level of .05, the mean difference = 5, standard deviation = 25, and the effect size = .20, for the differences between two groups of interest (e.g., borrowers with two different delivery modes of participation in training/technical assistance) in the outcome measure of pre-post change in the number of business employees (Appendix D-1). The mean difference and standard deviation used in these assumptions were based on a previous small business evaluation conducted by Optimal for the SBA. As detailed in the previous section, the sample size of 820 results in 55 cases per 12 strata and 80 cases per two West strata allowing for a national and regional representation.
The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.
The following data collection techniques will be used to maximize the survey response rate and minimize respondent burden.
Use the SBA official introduction email to intermediaries to establish the legitimacy and the importance of the survey (Appendix C-3). The introduction will describe the study’s purpose and value, the privacy and confidentiality of responses, and their use in the analysis.
Use the intermediary introduction email to borrowers to make them aware of the survey’s importance and legitimacy (Appendix B-3).
Send up to five email reminders at various times and on various days of the week to increase the survey invitation's odds of being noticed and acknowledged (Appendixes B-6 and C-6).
Conduct up to three reminder telephone calls with non-respondents (Appendixes B-5 and C-5). The respondents will have the option of taking the web survey or identifying another appropriate point of contact.
To maintain the sampling design's integrity, minimize biasing the sample, and ensure reliable and nationally representative results, Optimal will conduct up to 9 contact attempts with non-respondents via both email and phone follow-ups before replacing them. Thus, if no contact is made with a respondent after 9 email and phone call attempts or he/she does not want to participate, then Optimal will replace this respondent with another sampled borrower with the same strata specifications. This approach will ensure the accuracy and reliability of information collected to provide nationally representative results for the borrowers’ characteristics, experiences with the program, training and technical assistance received, and the business growth outcomes. Data collection efforts target the full census of intermediary lenders. Therefore, no sampling strata is used, and the efforts to maximize the survey response rate and minimize non-response will follow the schedule outlined in Exhibit 1.
Non-sampling errors arising from unit non-responses will be dealt with through weighting. The non-response bias analysis will be conducted to identify significant differences between respondents and non-respondents with respect to the data elements available in the administrative data. The significant differences will be dealt by developing nonresponse weights. Post-stratification weights will be used so that the weighted totals within mutually exclusive cells equal the known population totals.
Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from ten or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
Web survey and telephone interview pretesting were conducted with eight intermediaries and six borrowers. The pretesting was done via a modified version of cognitive testing. The cognitive testing examines four stages of the question-response process: comprehension of the question, retrieval/memory issues, formulation of the response, and the actual response (the fit between the response and the survey response categories). During these stages, information is obtained regarding the degree of difficulty respondents experience as they formulate an accurate response to the question. A slight modification to cognitive pretesting, called respondent debriefing, was implemented. This approach uses quick follow-up probes to check if respondents are having difficulty understanding the meaning of the questions or words, problems remembering the information needed to answer questions, issues with unnecessary questions or response options, or missing questions or response categories. The detailed probes for the most difficult items are administered at the end of the survey to minimize interference with the survey administration process. The respondent debriefing approach is useful for pretesting self-administered surveys as it approximates actual data collection procedures and provides an estimate of the time required to complete the survey and the respondent burden.
The items reported by the respondents or noticed by the interviewers as being problematic, difficult, or time-consuming to answer were discussed after the survey completion to determine the reasons for the difficulties. The interviewers also noted items that were difficult to recall and survey structure issues related to the layout, navigation, order of the questions, and the format of the survey.
On average, the borrower web surveys took about 20 minutes to complete and the intermediary survey took about 15 minutes. The borrower and intermediary lender interviews required about 20 minutes to complete, even with probing and discussions about the problematic questions. The estimated burden per each data collection material is provided in Exhibit 1, section 12.
Most of the interview participants provided detailed information in response to the questions and typically, there was no need for probes or follow-up questions. Overall, the pretest participants were very cooperative and often provided detailed feedback about their experiences with the program and reasons for having issues and difficulties with some of the survey questions. Based on the pretesting results, the surveys were revised to minimize the respondents’ burden. The revisions involved: rewording questions to improve comprehension and responding; omit confusing, time-consuming, or difficult to answer items; adding skip question options to avoid asking questions that were not applicable; and adding a few specific questions and items. The final versions of the surveys appear in Appendixes B-1 and C-1.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The consultants used to design, collect and analyze the information for the SBA are from Optimal Solutions Group, LLC. The contact information for the Optimal research team is below.
Program evaluation/labor economist |
Mark Turner, Ph.D. |
CEO and President |
Optimal Solutions Group, LLC |
|
Program evaluation, survey methodology |
Harrison Greene, ABD |
Project Director |
Optimal Solutions Group, LLC |
|
Sampling, survey methodology |
Michael Costello, M.S. |
Statistician |
Optimal Solutions Group, LLC |
|
Program evaluation, survey methodology |
Andrey Vinokurov, Ph.D. |
Task lead |
Optimal Solutions Group, LLC |
For questions regarding the study or questionnaire design or statistical methodology, contact Shay Meinzer, Lead Program Evaluator, [email protected], 202-539-1429.
1 http://www.payscale.com/research/US/Job=Small_Business_Owner_%2F_Operator/Salary
2 https://www.bls.gov/oes/current/oes111021.htm; https://www.bls.gov/oes/current/oes_nat.htm
3 https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/20Tables/html/DCB.aspx
4 West Region Rural strata were small and had to be combined with West Urban strata, for a total of 14 strata.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kelsye Turner |
| File Modified | 0000-00-00 |
| File Created | 2021-01-16 |