ScaleUp Supporting Statement Revised OMB Feedback 12-14-15

Appendix C-1 Request for Administrator Distribution of Participant Survey Announcement (and accompanying announcement)
Appendix C-2 Participant Survey Invite E-Mail
Appendix C-3 Participant Survey Reminder
Appendix C-4 Participant Survey Thank You E-Mail
Appendix C-5 Comparison Group Member Recruitment E-mail from SBA
Appendix C-6 Comparison Group Member Recruitment Screening Tool
Appendix C-7 Comparison Group Screening Tool Reminder E-mail
Appendix C-8 Comparison Group Survey Invite E-mail
Appendix C-9 Comparison Group Survey Reminder
Appendix C-10 Comparison Group Survey Thank You E-mail

PART A. Justification

Circumstances Making the Collection of Information Necessary

Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

ScaleUp America, a new initiative managed within the Small Business Administration’s (SBA’s) Office of Entrepreneurial Development (OED), is designed to expand the delivery of proven best practices in entrepreneurship education to reach more growth-oriented small business owners. Through this initiative, organizations in eight communities across the United States have been selected to deliver targeted and intensive assistance to established, growth-oriented small businesses and entrepreneurs. ScaleUp initiative goals include growing participating businesses, strengthening local entrepreneurial ecosystems (i.e., the network of supportive resources available to the entrepreneur, such as diverse suppliers, sources of capital, a “brain trust” of needed technical knowledge, skilled workers, and needed public infrastructure), and creating jobs and economic growth in targeted communities.

Currently, some data measuring program inputs and outputs are collected from the eight administrators of the program through quarterly or annual progress reports. However, there is no process in place to systematically collect participant outcomes. A centrally coordinated data collection effort is essential to comprehensively assess how ScaleUp operates at different sites and to measure the outcomes and impacts of ScaleUp on participant businesses.

SBA has hired an independent contractor, Optimal Solutions Group, LLC (Optimal), to conduct an impact evaluation of ScaleUp. The evaluation methodology includes administering a participant and comparison group member intake survey, and participant and comparison group member follow-up outcome surveys which together will provide a reliable quasi-experimental design to evaluate program impact. The survey instruments and research design are partly in response to the Office of Management and Budget (OMB) Memoranda M-13-17 and OMB Memorandum M-14-06 requirements for evidence-based evaluations and data that is “linkable” to federal administrative data.

Purpose and Use of the Information Collected

Indicate how, by whom, how frequently, and for what purpose, the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

This is a new collection effort. The ScaleUp initiative is a cohort, classroom-based program that operates two sessions (i.e. two cohorts) per year. For the ScaleUp impact evaluation, SBA has developed a total of five new data collection tools to be implemented each year, including 1) a participant intake survey at the start of the program, 2) a participant interview protocol at the end of the program, 3) a participant follow-up survey to be administered annually for two years after program completion, 4) a comparison group member “intake” survey, and 5) a comparison group follow-up survey to be administered annually for two years, and. One intake survey and two (identical) follow-up surveys will be administered to program participants and a group of recruited comparison firms over the course of three years, completing one survey per year.

The ultimate goal of this evaluation is to determine if the ScaleUp initiative generates greater business growth among participants than would have occurred otherwise. Additionally, the proposed data collection will be used to examine how to better deliver ScaleUp initiative services to traditionally underserved communities and populations, how to strengthen small business ecosystems in the communities, and to monitor performance indicators of the program.

The data collected will be used in three ways. The first is to report on the effectiveness of the program using a quasi-experimental research design to measure key business outcomes such as revenue and employment growth. A second purpose is to evaluate perceptions and other qualitative data that may suggest why the program is or is not effective. This may include such analyses such as determining whether small businesses perceive changes in the local assistance “ecosystem” over time and comparing perceptions to quantitative measures of the ecosystem obtained from publically available data (Stangler and Bell-Masterson 2015). A third purpose is to present annual performance data on ScaleUp using metrics that are common across multiple SBA programs. Unlike the first purpose, performance reporting does not inform whether the program is effective, but it does provide trend information that is comparable across programs and engages stakeholders with anecdotal evidence of what “success” looks like in ScaleUp.

The major research topics covered by the surveys and participant interviews are presented in Exhibit 1. Survey and interview instruments are in Appendices B-3, B-4, B-7, B-8, and B-10.

Exhibit 1. Instrument Research Topics

Research Topics			Source of Information
Research Topics	Participant Intake Survey	Comparison Group Member Intake Survey	Participant Follow-Up Surveys	Comparison Group Follow-Up Surveys	Participant Interview
Owner demographics	x	x
Business stage and unique identifiers	x	x
Business goals and needs	x	x	x	x	x
Business characteristics	x	x
Number of employees	x	x	x	x	x
Business revenue	x	x	x	x	x
Contracts and capital	x	x	x	x	x
Business challenges	x	x	x	x	x
Assistance received (from ScaleUp or elsewhere)			x	x	x
Participant satisfaction			x		x
Perception of community’s small business ecosystem	x	x	x	x	x

Use of Information Technology and Burden Reduction

Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden. Insert any applicable electronic web address.

SBA is committed to complying with the E-Government Act of 2002 to promote the use of technology in order to lessen the burden of data collection. Survey data will be collected via web-based surveys, which facilitate the efficient collection and analysis of data Web-based surveys reduce the burden placed on respondents because the survey platform ensures respondents do not need to read through instructions so follow skip patterns for questions that are not relevant to them because the skip pattern is automated. The web-based survey requires no more than hitting a submit button to transfer the information. The platform selected will also allow respondents to stop at any time and resume the survey at the point at which they last stopped

To the extent possible, follow-up surveys will be pre-populated with the static information provided at intake and confirmed rather than entered anew. Note that in most cases (e.g. the businesses zip code), it is important to give respondents the chance to update the information if it was incorrect at the source or has changed over the year, so these questions cannot be eliminated. Notices regarding the surveys and interviews, as well as the consent forms, interview protocols and other information related to this collection will be sent electronically to help minimize the burden on respondents.

Efforts to Identify Duplication and Use of Similar Information

Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

SBA does not currently require baseline information or follow-up information about ScaleUp participants and their perceptions of the small business ecosystem in their community. Both the participant and comparison group member surveys will be the first of their kind, collecting information that allows for a rigorous evaluation of the program. The economic data items to be collected via the survey, though potentially collected by a federal agency, are not currently available in any public administrative data that could be linked to ScaleUp participant data. Nonetheless, the survey instruments will collect business identification numbers and SBA is pursuing ways to obtain confidential, federal administrative data to reduce participant burden in future years of the program or in extended follow-up analysis on current participants.

Impacts on Small Businesses or Other Small Entities

If the collection of information impacts small business or other small entities, describe any methods used to minimize burden.

This data collection will obtain information from small businesses participating in the ScaleUp initiative. Additionally, in order to apply a quasi-experimental method in the impact evaluation, a comparison group of similarly oriented small businesses will also be recruited to complete the survey. To minimize the burden, efforts have been made to simplify and streamline the participant and comparison group member intake and follow-up surveys. The estimated burden is small, a maximum of 938 hours per year, with very few open-ended questions in the surveys. Both intake and follow-up surveys will be disseminated via an online platform, which will allow small business owners to take the survey at a time that is convenient, exit the survey if necessary and resume from the point that they last stopped, and only answer questions that pertain to their experiences. To the extent possible, follow-up surveys will be pre-populated with the static information provided at intake and confirmed rather than entered anew. More extensive data will only be gathered from a small subset of participants (i.e. 16) through interviews.

Consequences of Collecting the Information Less Frequently

Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

Data will be collected from the ScaleUp sites’ two annual cohorts. Each cohort will be asked to complete an intake survey and two follow-up surveys over the course of three years (i.e. one survey per year). One intake and two follow-up surveys will also be administered to a group of comparison firms that have not participated in the program. Although there are no technical or legal obstacles to reducing the burden, it is necessary to measure intake and outcomes in two cohorts and for two years to obtain reliable statistics on business growth, as described below.

Two years of follow-up is needed because it is adequate time to both smooth anomalies in annual business cycles and to more adequately realize the growth gains attributable to program participation. Small businesses often experience transitory changes, such as using temporary work contracts or winning an unusually large contract in one year. Additionally, it can take more than a year for a participant to enact its growth plan if the plan requires acquiring new capital or breaking into a new market. Thus one year of follow-up is not adequate to describe the typical post-program business growth. Two cohort’s worth of data is required because the cohorts are small and unlikely to produce statistically significant results without combining them.

The standardized data collection process proposed here is part of a framework that will allow SBA to conduct a quasi-experimental impact evaluation of the program. If both participant and comparison group member intake and follow-up surveys are not conducted, SBA will lack the data that is needed to compare change before and after the program between the treatment and comparison group, elements necessary for a quasi-experimental research design.

ScaleUp participant interviews will only be held once with up to two participants in each of the eight ScaleUp communities. The qualitative data generated during interviews will contextualize and help with the interpretation of the quantitative results from the participant and comparison group member surveys. If SBA is unable to conduct participant interviews, it will not be able to fully assess the individual variation in the ScaleUp assistance process, the best practices, challenges, lessons learned, and the distinctive cultures in different ScaleUp programs that might distinguish them from other similar business assistance programs.

The above described 3-year impact study will be conducted annually. The first study will follow cohorts 1 and 2 from intake through two follow-ups, the second study will follow cohorts 3 and 4 from intake through two follow-ups, the third study will follow cohorts 5 and 6, and so on. It is important to conduct the impact study for at least three new program cycles. The additional data is needed to ensure impact results represent the fully implemented program after implementation adjustments in the early years. A new comparison group is required for each year because the initiative targets businesses of a certain size and age. The first comparison group may exceed the eligibility requirements in the second year and thus not be comparable to the new year of participants.

Special Circumstances Relating to the Guideline of 5 CFR 1320.5

Explain any special circumstances that would cause an information collection to be conducted in a manner:

requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary, trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

SBA is requesting approval to provide less than 30 days for the participants to complete their intake surveys. Participants will receive the intake survey at the start of their classes. It is important that they document their behaviors and perceptions before course content is delivered so the results can be compared to the post-course follow-up assessment and meaningful differences can be detected. There are no other special circumstances that would cause information to be collected in the manner described above.

Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.

The required public comment notice was published in the Federal Register on May 1, 2015 at 80 FR 24998 (see Appendix A-3). All comments were due on or before June 30, 2015. No comments on this collection were received.

Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.

Apart from the request for comment through the FR notice, the only other consultation outside of the agency was with Optimal, the company SBA has contracted with to design the methodological approach for data collection and the survey instruments. Optimal functions as an independent third party to provide recommendations on the deliverables. Optimal also conducted pilot tests of all instruments and refined instructions and questions for clarity as needed. Those changes are reflected in this proposed survey. Optimal’s team is composed of researchers with experience in diverse disciplines, including sociology, labor economics, and program evaluation. The contact information for these individuals is presented in Section B of this document.

Explanation of Any Payment or Gift to the Respondents

Explain any decision to provide any payment or gift to respondents, other than re-enumeration of contractors or grantees.

All members of the comparison group will be offered $50 in compensation (in the form of a VISA gift card) in return for completing each survey instrument (for a total of $150). This is offered in order to promote a high response rate among comparison group members, thereby contributing to the integrity of the quasi-experimental evaluation design. This compensation (equivalent to about $100 per hour) is deemed necessary to motivate response given that comparison group members will have received no free services and as small business owners are likely to perceive their time as very valuable.

Offering financial compensation to increase survey response is supported by a large literature, which has also shown its limited effect on the quality of the data (e.g. Coughlin et al., 2011). Most recently, the Department of Labor achieved response rate of 80 percent or more for comparison group members using such an incentive structure in a similar entrepreneur training program and quasi-experimental research design (Michaelides and Benus, 2012). As SBA’s first quasi-experimental evaluation study in recent years, a viable comparison group is vital to a successful investment in the Federal government’s efforts to promote evidence-based evaluation.

Assurance of Confidentiality Provided to Respondents

Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

Optimal, an independent contractor, will be collecting, compiling, and analyzing all data collected from ScaleUp participants and the comparison group members. Optimal was hired to ensure that the information provided by clients will not compromise their receipt of services. The data will be collected through and stored on Optimal’s secure servers. Optimal implements administrative, physical and technical controls to protect personal and sensitive information. Secure intranets are used to maintain project-related files, and secure servers use industry-standard methods such as firewalls, monitored access logs, virus protection, encrypted connections, password-protected accounts, and user authentication mechanisms to ensure the privacy of personal data. Optimal maintains a biometrically (physically) secure environment and employs a data security officer who oversees Optimal’s data. The security approaches are further described in Appendix A-2.

In the study reports produced for the public, participants and comparison group member data will be reported in the aggregate and information obtained from the interviews will be attributed to anonymous participants. The information is subject to the protections of the Privacy Act of 1974, and SBA will not disclose information under the Freedom of Information Act if that information is shown to be a trade secret or information that is commercial or financial in nature and privileged or confidential. As stated in the contract with Optimal, contractors are required to comply with the requirements of the Privacy Act when they are maintaining a system of records on behalf of SBA. Participants are informed of these conditions in the consent forms seen in Appendices B-2 and B-6.

Justification for Sensitive Questions

Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

ScaleUp participants and comparison group members will be asked about their businesses’ income, profits, loans, and contract awards. An essential component of the ScaleUp service is to provide assistance with gaining access to capital and growing their business through contracts and more. Therefore, the participants and comparison group members will be asked about their business’s financial situation before and after the program to assess change. Although potentially sensitive in nature, questions regarding business performance are essential in determining whether ScaleUp is having its intended impact. The information will be used to report on growth of participants, relative to comparison firms. Participants and the comparison group members will be provided with this background information in the e-mail invitation (see Appendix C-1) and their consent will be requested in the survey invitation.

Participant and comparison group member intake survey instruments also include questions about race, ethnicity, veteran status, and disability in order to determine whether ScaleUp is fulfilling its mission of serving service-disabled and other socioeconomically disadvantaged entrepreneurs. These questions were written in compliance with the OMB Standards for the Classification of Federal Data on Race and Ethnicity and largely mirror the demographic questions on OMB-approved SBA Form 641.¹ Survey respondents are allowed to select “prefer not to answer” to these question.

Participant and comparison group members will also be asked for their business identification numbers (i.e. EIN, DUNS). These data are necessary to improve data validity and reduce survey burden in future years of the study or in extended follow-up analysis on current participants. SBA is pursuing access to federal datasets at the Internal Revenue Service and Census Bureau, which maintain economic data on businesses that is linkable via EIN and DUNS, respectively. Additionally, unique identifiers will allow SBA to link ScaleUp participants to the data it houses on loans and contracts. Given the sensitive nature of these identifiers, EIN in particular, the data will be transferred and maintained according to a data security protocol (see Appendix A-2).

Estimates of Hour Burden, Including Annualized Hourly Costs

Provide estimates of the hour burden of the collection of information. Indicate the affected public, number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.

SBA estimates that 2071 hours are needed to conduct this study over the course of the three years. The maximum for any one year is 938. The total annualized hour and costs burdens are set out below in Exhibit 2 and 3.² The affected public and respondent types for this data collection are composed of ScaleUp administrators (i.e. the contract awardees), small business participants, comparison group member respondents, and comparison group member recruits. Preliminary pre-tests were conducted for all data collection instruments to derive the burden estimates.

Exhibit 2. Study burden estimate for each cohort set (e.g. cohorts 1 and 2)

Burden	Number of small businesses	Frequency of collection	Average minutes spent	Small business burden (in hours)	Small business burden (in dollars)	Administrative burden for 8 sites (in hours)*	Total small business and administrator burden (in hours)
Participant interviews
Recruitment and scheduling email	16	1	5	1	33	2	3
Interview	16	1	30	8	200	0	8
Intake surveys, including comparison group recruitment
Survey introduction email to participants	272	1	2	9	227	4	13
Survey introduction email to comparison group	2000	1	2	67	1,667	0	67
Survey screening questions (comparison only)	400	1	6	40	1,000	0	40
Survey invitation email	592	1	2	20	493	0	20
Survey reminder emails/calls	592	4	1	39	987	4	43
Survey completion	592	1	20	197	4,933	0	197
Survey thank you email	592	1	1	10	247	0	10
Annual burden CY 1				391	9,787	10	401
1st follow-up survey
Survey invitation email	592	1	2	20	493	0	20
Survey reminder emails/calls	592	4	1	39	987	2	41
Survey completion	592	1	20	197	4,933	0	197
Survey thank you email	592	1	1	10	247	0	10
Annual burden CY 2				266	6,660	2	268
2nd follow-up survey
Survey invitation email	592	1	2	20	493	0	20
Survey reminder emails/calls	592	4	1	39	987	2	41
Survey completion	592	1	20	197	4,933	0	197
Survey thank you email	592	1	1	10	247	0	10
Annual burden CY 3				266	6,660	2	268
Total over 3 years

*There are no additional costs associated with the administrators as they are performing these functions under an existing contract.

Exhibit 3. Annualized study burden estimate over three years

	Respondents	Year 1 burden	Year 2 burden	Year 3 burden	Total burden hrs.
Cohort 1 and comparison group	456	333	207	207
Cohort 2	136	130	0	62
Subtotal over 3 years	592	463	207	268	938
Cohort 3 and comparison group	456		333	207
Cohort 4	136		130	0
Subtotal over 2 years	592		463	207	670
Cohort 5 and comparison group	456			333
Cohort 6	136			130
Subtotal over 1 year	592			463	463
Total burden hours required per year		463	670	938	2071

Total respondents in one year		592	1048	1640
Total responses in one year		744	1200	1792

The intake and follow-up surveys will be sent to two cohorts of ScaleUp participants. Each of the cohorts consists of approximately 136 program participants across all sites. Additionally, 250 individuals in each of the eight ScaleUp communities (a total of 2,000) will be contacted for recruitment into the comparison group. It is estimated that 400 (20 percent) of the 2,000 will voluntarily complete the screening survey and that 320 (80 percent) will be eligible for the study.

These estimates assume a 100 percent response rate, although the actual response rate will likely be lower (see part B of this supporting statement). The comparison group member screening questions and all associated emails for the surveys are included in Appendices C-5, C-6 and C-7.

Based on the burden test, the estimated average completion time of an intake survey is 20 minutes and follow-up survey is 20 minutes. Reading material related to the survey is estimated at 1 to 6 minutes depending on the material, as shown in the table.

Two ScaleUp participants will be selected in each of the eight ScaleUp communities to participate in a semi-structured 30-minute interview regarding participant and community successes attributable to ScaleUp activities. The total estimated annual hour burden, including contact and scheduling time for respondents and administrators, is 11 hours.

13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information.

There is no other total annual cost burden to respondents or record keepers.

14. Annualized Costs to Federal Government

Provide estimates of annualized cost to the federal government. Provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.

The annualized cost to the federal government is $409,880 (see exhibit 4). This includes the costs associated with the contractor conducting the project, the comparison group payment, and the salaries of the assigned SBA OED employees.

Exhibit 4. Annualized costs to the federal government

Estimates	SBA COR	SBA advisor	Evaluation contractor	Payment to comparison group	Total federal costs
Number of employees	1	1	NA	320	NA
Hours anticipated	80	80	NA	NA	NA
Average hourly pay	$43.67	$51.60	NA	NA	NA
Total cost	$3,494	$4,128	$384,000	$16,000	$407,622

The cost of the OED employee, Research Associate, involved in administration of the survey and study is estimated at GS-13, Step 1, at $43.67 per hour based on 2,080 hours per year. OED anticipates that this person will work 80 hours per year for 1 year. The annual cost for this OED employee over the course of this study is $3,494. The cost of the OED employee involved in study oversight is estimated at GS-14, Step 1, at $51.60 per hour based on 2,080 hours per year. OED anticipates that this person will work 80 hours per year for 1 year. The annual cost for this OED employee over the course of this study is $4,128. The annual cost for both of these OED employees over the course of this study is $7,621.08. Federal employee pay rates are based on the General Schedule of the Office of Personnel Management for 2015 for the Washington DC locality.³

The cost of ScaleUp administrators is based on an hourly salary of $31.36, which is from the Bureau of Labor Statistics Occupational Employment and Wages. It is the mean hourly wage of 11-9151 Social and Community Service Managers as of the May 2013 release. Comparison group members (320) will be paid $50 to complete each survey for a total of $16,000 in costs to the government. All costs for conducting the study are included in the contract between SBA OED and Optimal under the contract number SBAHQ-13-D-0003 and equal $384,000. This encompasses the finalization of the evaluation architecture, data-collection instruments, the Federal Register listing, and the OMB package, as well as the data collection and production of an impact evaluation report.

15. Explanation for Program Changes or Adjustments

Explain the reasons for any program changes or adjustments reported in items 13 or 14 of OMB form 83-I.

This is a new data collection; there are no program changes or adjustments.

16. Plans for Tabulation and Publication and Project Time Schedule

For collections of information whose results are planned to be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

This study will use both quantitative (i.e., descriptive and inferential statistical techniques) and qualitative methods for analyzing the data. The primary statistical analysis technique planned is a regression-adjusted difference-in-difference estimation. This approach involves calculating differences between a treatment and a comparison group of subjects (the first difference) in the changes in key variables over time (the second difference). See exhibit 4, where the variable R_it denotes an outcome measure at a point in time (such as revenue), subscript i is an indicator for either the treatment group (“a”) or the comparison group (“b”), and subscript t indicates the time period.

Exhibit 4. Difference-in-differences construction for treatment and control groups
Time Period	Selected Program Applicants (Treatment group, "a")	Unselected Program Applicants (Comparison group, "b")	Difference
2	R_a2	R_b2	(R_a2 – R_b2)
1	R_a1	R_b1	(R_a1 – R_b1)
Change	(R_a2 – R_a1)	(R_b2– R_b1)	(R_a1 – R_b1) – (R_a2 – R_b2)

The analysis is expected to include tables for the statistics and potentially graphs. They will include the appropriate associated statistics, such as the p-value for statistical significance at the .1, .05 and .01 levels. Other data gathered from the surveys and used in the impact analysis will be presented in tables using descriptive statistics. Qualitative analysis will involve determining challenges, lessons and the factors participants consider contributed to their success. These qualitative analyses will be presented in a distinct way from the impact analysis. A final public report will be released approximately three years after the start of the program year. Additionally, an annual ScaleUp performance report will be produced that provides descriptive statistics on inputs and outputs of the program, as well as showcases growth milestones achieved by participants interviewed and provides anecdotal “success stories” to engage readers. A standardized schedule for the project is attached in the Appendix A-1.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

SBA plans to display the OMB expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

Explain each exception to the certification statement identified in item 19, “Certification Requirement for Paperwork Reduction Act” of OMB Form 83-I. If Agency is not requesting an exception, the standard statement should be used.

SBA is not requesting any exceptions to the certification.

References

Coughlin, Steven S., Aliaga, P., Barth, S., Eber, S., Maillard, J., Mahan, Clare M., Kang, Han K., Schneiderman, A., DeBakey, S., Vanderwolf, P. and Williams, M. (2011) “The Effectiveness of a Monetary Incentive on Response Rates in a Survey of Recent U.S. Veterans.” Survey Practice 4(1).

Michaelides, M., and Benus, J. (April 2012). Are Self-Employment Training Programs Effective? Evidence from Project GATE. IMPAQ International.

Strangler, Dane and Jordan Bell-Masterson. (2015). Measuring an Entrepreneurial Ecosystem. Kauffman Foundation.

PART B. Collections of Information Employing Statistical Methods

Universe and Sampling Respondent Selection

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The intake and follow-up instruments will be sent to all ScaleUp participants in Cohorts 1 and 2 at the eight sites. The participant universe is 272 small businesses that received ScaleUp assistance during fiscal year 2015. A sampling strategy will not be used due to the small size of the cohorts. The universe of comparable small businesses that are not participating in the initiative at each site is unknown. The goal is to identify 2,000 potentially comparable businesses and after screening and nonresponse, maintain a comparison group of about 270 businesses. It is estimated that 20 percent of businesses identified will respond to the screening questions (400) and 80 percent of those businesses will be eligible to participate (320).

The comparison group recruitment and selection strategy is built around the need for a minimum number of response units to detect statistically significant differences from the participants, maintaining approximately comparably sized groups between the participants and comparison members across all sites, and limiting the degree of bias introduced in quasi-experimental design that does not rely on random assignment to control and treatment group. An initial list of 2,000 small businesses will be identified from local (e.g. Chambers of Commerce) and national (e.g. Hoovers) small business directories. Selection of the 2,000 will be based on comparability to the ScaleUp participants’ business revenues, business ages and geographic locations. The directories currently under consideration have a minimum of two, and in some cases, all three criteria. Each potential comparison group business will be contacted and provided a screening survey to verify the business information obtained in the directory and to further assess comparability to ScaleUp participants through such questions as their interest in growing their business and their industry (see appendix C-6). From the estimated 400 businesses that respond to the screening survey, 320 will be selected with a goal to maximize comparability, reduce bias, and maintain a minimum number of businesses in the comparison group.

Exhibit 1 details the expected sample sizes for the participant and comparison group. Because matched intake and follow-up surveys are required for the difference-in-difference analysis planned, the follow-up survey response rate is shown in the table. The response rate estimate is based on a similar Department of Labor study that used an incentive for its comparison group (Michaelides and Benus 2012) and achieved an 80 percent response rate to the first outcome survey. An 80 percent response from “un-incentivized” participants is higher than is often achieved in business assistance outcome surveys.⁴ However, the cohort design of the ScaleUp initiative ensures more regular contact between participants and administrators. Moreover, an intentional awareness strategy was developed at the start of the ScaleUp initiative. This strategy informed ScaleUp administrators of the types and purpose of the data collections so they could inform participants very early on in his/her acceptance to the program. See question 4 for other strategies that will be used to obtain the response rates shown in exhibit 1.

Exhibit 1: Expected Response Rate by Survey Population

Type of respondent	Universe	Expected number of respondents	Expected Response Rate
Participant Surveys	272	218	80%
Comparison Group Surveys	320	256	80%

Describe the Procedures for the Collection of Information

Statistical methodology for stratification and sample selection, estimation procedure, degree of accuracy needed for the purpose described in the justification, unusual problems requiring specialized sampling procedures, and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

As noted in question 1, no sampling strategy is used in this data collection. Data will be collected from all ScaleUp participants as well as from a matched comparison group of small business owners in the same community. Three types of statistical analyses will be conducted, descriptive statistics (e.g. mean, standard deviation), Heckman Selection Correction (HSC) and other efforts to control for differences in the comparison and control group, and a regression adjusted difference-in-difference estimation (D-in-D) to evaluate program effectiveness. Exhibit 2 shows the research topics introduced in Part A of this packet, with the type of quantitative analysis planned for each. Note that data sources such as participant interviews, and research topics such as perceptions of the small business ecosystem, will be analyzed qualitatively to identify themes and not used to make claims about the effectiveness of the initiative. Estimates from the statistical analysis will be generalizable to the eight ScaleUp sites, which were selected purposefully to fulfill predefined SBA contract criteria, but will not necessarily be representative of other regions of the U.S. or other locations in which the initiative may be deployed in the future. The primary objective of the impact evaluation is to provide statistically valid and reliable estimates of the incremental effects of ScaleUp services on key business outcomes, including revenue and employment growth. The entire program rather than the individual sites are the unit of analysis for the statistical methods employed.

Exhibit 2: Research Topics, Data Sources and Statistical Analyses

Research Topics	Source of Information
Research Topics	Participant Intake Survey	Comparison Group Member Intake Survey	Participant Follow-Up Surveys	Comparison Group Follow-Up Surveys	Participant Interview	Statistical Analyses
Owner demographics	x	x			NA	-Descriptive -HSC
Business stage	x	x			NA	-Descriptive -HSC
Business goals and needs	x	x	x	x	NA	-Descriptive (potentially)
Business characteristics	x	x			NA	-Descriptive -HSC
Number of employees	x	x	x	x	NA	-Descriptive -HSC -D-in-D
Business revenue	x	x	x	x	NA	-Descriptive -HSC -D-in-D
Contracts and capital	x	x	x	x	NA	-Descriptive -HSC -D-in-D
Business challenges	x	x	x	x	NA	-Descriptive (potentially)
Assistance received (from ScaleUp or elsewhere)			x	x	NA	-Descriptive -HSC
Participant satisfaction			x		NA	-Descriptive
Perception of community’s small business ecosystem	x	x	x	x	NA	-Descriptive (potentially)

The primary analysis method used to measure program effectiveness is regression adjusted difference-in-difference estimation, which allows for the estimation of differences between participants and comparison group members (the first difference) in the changes over time in key variables (the second difference). The changes over time in key small business outcomes (e.g. small business revenue or employment) that are attributable to participation in ScaleUp are estimated, controlling for time period, program participation, location, time-variant community economic characteristics (e.g. local unemployment rates), and time-variant small business characteristics. Observed differences collected in the surveys between the participant and comparison group will also be used as controls in the regression. Estimated standard errors are adjusted appropriately for grouping of data by site. Standard inferential statistics will be employed to determine if estimated differences are statistically significant, using a standard threshold of a 5 percent significance level to reject the null hypothesis of no difference.

Double-differencing has the effect of helping control for selection issues by eliminating the influence of all observed and unobserved time-invariant characteristics (that have time-invariant effects on the outcome in question). Additionally, HSC will be employed to reduce selection bias in the model estimates caused by pre-existing differences in the comparison and treatment group. Business and demographic characteristics not screened during the comparison group selection and that may relate to the probability of joining the ScaleUp program include factors such as the owner’s race and business structure. To the extent that these factors are correlated with unobserved influences on program participation (e.g. interest, growth potential), the method will help reduce bias due to self-selection. The individual predicted probabilities of selection determined in the HSC model will be used as control variables in the difference-in-difference estimation. Although this technique helps to mitigate selection bias, countless observed and unobserved characteristics could influence a small business’s likelihood of participating in ScaleUp and thereby confound the results of a quasi-experimental study. A superior method of comparison group member selection would be to recruit ScaleUp applicants that the sites had to turn away due to the limited number of spots available. As of the first year of the program, such oversubscription was not found among the sites and therefore is not proposed in the current study design.

If the follow-up response rate falls below the expected 80 percent, non-response bias analysis will be conducted using demographic and business information collected in the intake survey. The analysis will consist of the Student’s T test for continuous variables such as business revenue and Chi-squared statistics for categorical information such as the owner’s industry. The report will detail how differences in these observable factors among respondents are likely to effect the estimates derived. The potential bias in outcome estimates due to selection bias and non-response will be described in the interim and final reports to increase transparency of the study.

Describe Methods to Maximize Response Rates and methods to Deal with Issues of Non-Response

The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

To maximize the follow-up survey response rate and minimize respondent burden, the following data collection techniques will be used.⁵ Surveys will be introduced to the cohort participant and comparison group members via an official introduction e-mail from the ScaleUp initiative director (or SBA in the case of the comparison group). This will help to legitimize the survey and highlight its importance. The introduction outlines potential benefits to the business community resulting from the evaluation. It highlights that the survey will directly contribute to finding ways to further improve the program so that the needs of growth-oriented small business owners and entrepreneurs are better served. It also explains the confidentiality of the information provided by the respondent to decrease non-response that results from concerns over data or contact information sharing.

After the introduction letters, Optimal will take a number of steps to maintain the visibility and immediacy of the survey. Optimal will send an e-mail to cohort participants and comparison group members with the direct survey link embedded for easy access. The surveys are designed to use skip patterns and simple, often multiple-choice answer options to facilitate the ease with which the survey can be completed once started. Optimal will send up to three reminder e-mails encouraging potential survey respondents to complete the survey before the deadline. The reminders will be sent at different times of the day to increase the probability of reaching respondents at an opportune time. Lastly, Optimal will make phone calls to non-respondents to encourage their participation should response rates fall below 80 percent (see appendix C-9). Phone calls will be particularly important during the second follow-up survey because ScaleUp participants may have little contact with the program after graduation.

Importantly, response rates would be extremely low for comparison group members because unlike participants, they have not received the free ScaleUp services. To mitigate this risk and reach an 80 percent response rate, the study will offer all comparison group members a $50 VISA gift card incentive for completing each survey, in addition to the steps above. This compensation is equivalent to about $100 per hour and is expected to generate high interest in completing the surveys. The incentive is higher than the $15 used in the DOL survey cited above. However, many of those study participants were unemployed and thus a smaller incentive may have been effective. Studies of the impact of financial incentives have found that larger incentives (e.g. $50 versus $20 or $25 versus $20) increased response rates significantly (Collins et al. 2000; Keating et al. 2008). During three pretest interviews for this study, small businesses were questioned about the size of a financial incentive and they indicated that $50 had just a small effect on their willingness to participate, therefore the $50 financial incentive was considered a minimum.

Describe Any Tests of Procedures or Methods to be Undertaken

Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from ten or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The participant and comparison group member intake surveys and the participant and comparison group member follow-up surveys and comparison group screening questionnaire were pretested to ensure reliability, minimize measurement error, and minimize respondent burden. The surveys were revised based on the feedback received from four invited respondents that are comparable in business types and size to the ScaleUp participants. Cognitive interviewing methods were used to review the instruments and items the respondent or the researcher noted were problematic, difficult, or time-consuming to answer, were further discussed to determine the reasons for the difficulties and ways to improve the question.

In addition, pretests on internal research staff were completed to provide a burden estimate for the final instruments. Staff were instructed to complete the survey in one sitting, at a time and place with minimal distractions. The longer of the intake and follow-up instrument was used for the estimate. The average burden across testers was as follows.

Exhibit 3: Burden estimates

Instrument	Average burden in minutes
Intake survey	17
Screening questionnaire	6

Expert Contact Information

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The consultants used to design, collect and/or analyze the information for the agency are from Optimal Solutions Group, LLC. The contact information for the Optimal research team is below.

Program Evaluation/ Labor economist	Laura Leete, Ph.D.	Senior Research Associate	Optimal Solutions Group, LLC	[email protected]
Program Evaluation/ Labor economist	Mark Turner, Ph.D.	CEO and President	Optimal Solutions Group, LLC	[email protected]
Program Evaluation	Jennifer Auer, Ph.D.	Project Director	Optimal Solutions Group, LLC	[email protected]

For questions regarding the study or questionnaire design or statistical methodology, contact the staff members listed above at:

Optimal Solutions Group, LLC

M Square Research Park

5825 University Research Court, Suite 2800

College Park, MD 20740-9998

Telephone: 301-918-7301;

E-mail: [email protected]

References

Collins, Rebecca L., et al. "Effects of incentive size and timing on response rates to a follow-up wave of a longitudinal mailed survey." Evaluation Review 2000: 347.

Keating, Nancy L., et al. "Randomized Trial of $20 versus $50 Incentives to Increase Physician Survey Response Rates." Medical Care 2008: 878.

Michaelides, M., and Benus, J. (April 2012). Are Self-Employment Training Programs Effective? Evidence from Project GATE. IMPAQ International.

1 Found at https://www.sba.gov/content/counseling-information-form.

2 The cost burden is based on an estimated median annual income of $66,500 for small business owners, which equates to an hourly rate of $25. See http://www.payscale.com/research/US/Job=Small_Business_Owner_%2F_Operator/Salary.

3 http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/15Tables/html/DCB.aspx

4 See examples at http://www.goldmansachs.com/s/10ksb-docs/29407_Book.pdf and https://www.sba.gov/sites/default/files/files/OED_ImpactReport_09302013_Final.pdf.

5 It is anticipated that the response rate to intake surveys will be close to 100%. Participants will be in weekly classes and administrators can prompt them to complete the survey. The comparison group will have just agreed to take part in the study in the previous weeks.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Kelsye Turner
File Modified	0000-00-00
File Created	2021-01-24

ScaleUp Supporting Statement Revised OMB Feedback 12-14-15