RIC PRA Supporting Statement 4-26-16

Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

The U.S. Small Business Administration’s (SBA) statutory mission is to "aid, counsel, assist and protect, insofar as is possible, the interests of small business concerns." The Agency’s Office of Entrepreneurial Development (OED) helps to carry out this mission by providing training and counseling programs and initiatives, such as the Regional Innovation Clusters (RIC) initiative, to existing and prospective small businesses.

Through RIC, SBA is investing in regional clusters—geographic concentrations of interconnected companies, specialized suppliers, academic institutions, service providers, and associated organizations with a specific industry focus—throughout the United States that span a variety of industries, ranging from energy and manufacturing to advanced defense technologies. The three primary goals of the initiative are to (1) increase opportunities for small business participation within clusters, (2) promote innovation in the industries on which the clusters are focused, and (3) enhance economic development and growth in cluster regions. This information collection is necessary for SBA to understand the progress of the RIC initiative toward achieving those goals.

Purpose and Use of the Information Collected

Indicate how, by whom, how frequently, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

In response to the Office of Management and Budget (OMB) Memorandum M-13-17, “Next Steps in the Evidence and Innovation Agenda,” and OMB Memorandum M-14-06, “Guidance for Providing and Using Administrative Data for Statistical Purposes,” SBA strives to make using data, including federal administrative data, a priority to inform decisions.

SBA plans to collect satisfaction and business economic data from program participants annually. Changes in key measures, such as revenue and employment growth, will be benchmarked against the average change in similar firms in the same regions. In addition, the participants’ satisfaction with the program and its effects on their short-term goals (and the goals of the Initiative) will be tracked from year to year. SBA has hired an independent contractor, Optimal Solutions Group, LLC (Optimal), to plan and conduct this performance evaluation of the RIC Initiative.

The RIC Initiative capitalizes on the theory of regional cluster development by supporting “actively managed” clusters (i.e. clusters that are administered by a team of individuals and possess a form of governance rather than a cluster that occurs naturally without intervention in a regional economy). The teams provide a host of services to the target population of small and emerging businesses within their regional and industry focuses. Services include direct business advising and support and sponsoring events, such as networking opportunities with investors, large businesses and other stakeholders in the regions.

The intended short- and intermediate-term outcomes are directly linked to cluster services, activities, and events, and are expected to be observed during or soon after participation in the RIC. Long-term outcomes, such as increased revenue and total payroll are expected to be observed in subsequent time periods.

Figure 1 illustrates the short-, intermediate-, and long-term outcomes that will be evaluated in this study, showing the linkages between cluster services and these outcomes as well as the metrics used to assess them.

Figure 1. RIC Logic Model

The data collection effort discussed in this request involves the collection of a variety of data from three classes of RIC initiative stakeholders. Small businesses participating in the cluster represent a first group of initiative stakeholders. They will be sent an online survey (or a PDF smart form, depending on their preference) to provide data about their participation and satisfaction, the performance of their firm with regard to a variety of key performance indicators that are expected to manifest themselves due to cluster participation, and the role of cluster participation in the evolution of these key performance indicators.

Large organizations, a broad group that includes universities and national laboratories, public sector agencies, nonprofit organizations, and business associations, represent the second group of initiative stakeholders. They will be asked to answer an online survey (or a PDF smart form, depending on their preference) to provide data about the reason(s) for their participation, their collaboration or involvement with small businesses, and the role of cluster participation on key indicators associated with their operations.

Finally, the administrators of the clusters involved in the RIC initiative will be asked to complete a survey that provides the frame from which small business and large organization surveys are sent, as well as information about the services they provided these two groups of stakeholders, and about their operations in general. In addition, cluster administrators will be interviewed twice a year, once to clarify and follow up on the content of their first through third quarter progress reports, and once to follow up on their fourth quarter progress report, to obtain information about how their operations have evolved, and the adjustments they made and the lessons learned during the year.

The proposed evaluation will be used to increase SBA’s understanding of outcome trends of small business across its RIC portfolio, identify potential areas of exception, and contribute to the body of knowledge surrounding how RICs help small businesses. This data will not be used to evaluate the effectiveness of an individual cluster.

The survey results do not allow for causal claims regarding the relationship between the outcome estimates and the services provided through the Initiative. The surveys capture outputs and outcomes of the small businesses participating in each cluster. At least three categories of users exist for the information collected and it is used in the following ways:

SBA uses the evaluation findings on a regular basis to monitor program trends across its portfolio and assess participants’ satisfaction with the services. Evaluation findings are also used by SBA during interactions with its stakeholders, including Congress.
Cluster administrators (who are provided clean, de-identified data for their cluster upon request) use this information to refine their strategy and improve the service mix they provide to participants. They may also use these data during discussions with potential partners, in an effort to obtain other sources of funding outside of SBA.
Researchers in the field of economic development and cluster practitioners use the findings from this evaluation (which are shared with the public) to implement or refine a cluster strategy in their region, improve how they measure their own initiative or project, or develop improved methods to measure cluster performance.

SBA and some of the cluster administrators use these data several times a year, whereas it is more difficult to assess the frequency of use of these data by researchers and practitioners who operate and evaluate clusters.

Use of Information Technology and Burden Reduction

Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden. Insert any applicable electronic web address.

SBA is committed to complying with the E-government Act of 2002 and OMB Memorandum M-03-22, “OMB Guidance for Implementing the Privacy Provisions of the E-government Act of 2002,” to promote the use of technology to lessen the burden of data collection. Survey data will be collected via web-based surveys and electronic templates, which facilitate the efficient collection and analysis of data. Web-based surveys reduce the burden placed on respondents, because the survey platform automates the skip pattern to ensure respondents do not need to read through instructions to bypass questions that are not relevant to them. The web-based survey requires no more than hitting a submit button to transfer the information. The platform selected will also allow respondents to stop at any time and resume the survey at the point at which they last stopped. The electronic templates are created in MS Excel and are programmed to operate with similar burden reduction methods as web-based surveys. For example, drop-down menus are used so that respondents do not have to type in the information, and skip patterns are implemented by “graying out” columns that are not applicable based on the responses to previous questions. In all surveys, a glossary of terms is provided, and in the case of the web-based survey, the glossary is hyperlinked to the instances of the term in the survey for quick reference. A glossary reduces confusion among respondents, thereby lessening their burden, and improves the consistency of the data collected by ensuring that respondents use the same definition.

Efforts to Identify Duplication and Use of Similar Information

Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

The surveys and cluster administrator interviews discussed in this packet were devised to fill gaps identified in the clusters’ progress reports. SBA does not currently require the systematic collection of baseline or follow-up information about RIC participants from the clusters. They are only required to provide quarterly reports where they are asked about their aggregated progress on a few key metrics. Therefore, there is no duplication in the data collected via the surveys and cluster administrator interviews.

There is only one area where duplication exists; the small businesses and the clusters in which they participate are both asked to report on access to external financing (e.g., venture or angel capital, SBA and non-SBA loans, grants). The underlying reasons for this are discussed in more detail in under question #5, below, as they relate to diminishing the burden on small businesses.

The economic data items to be collected via the survey, although potentially collected by a federal agency, are not currently available in any public administrative data that could be linked to RIC participant data. Nonetheless, the survey instruments will collect business identification numbers and SBA is pursuing ways to obtain confidential, federal administrative data to reduce participant burden in future years of the program or in extended follow-up analysis on current participants.

Impacts on Small Businesses or Other Small Entities

If the collection of information impacts small business or other small entities, describe any methods used to minimize burden.

To minimize the burden on respondents, which will include small businesses and small entities, efforts have been made to simplify and streamline the survey. The estimated annual burden on small businesses or small entities is only 228 hours (of the 388 total burden hours of the collection) per year, largely due to the small number of service satisfaction questions and business information that does not require the respondent to retrieve data from a secondary source, such as a tax return. The survey will be disseminated via an online platform, which will allow small business owners to take the survey at a time that is convenient for them and only answer questions that pertain to their experiences. For example, skip patterns are programmed into the platform so that businesses that reported no new access to external financing via a simple “yes” or “no” question are not presented with follow-up questions about the sources and total value of financing or the role played by the cluster in facilitating access to these sources. In the event that the small business is not comfortable sharing information in a web-based tool, they will be provided with a PDF smart form version of the survey that can be filled out by hand or electronically, as they prefer.¹

As mentioned in question #4 above, there is one instance where the same information is collected from two different groups of respondents: Both the small business survey and cluster administrator survey request information about access to external financing by small businesses participating in the clusters. This duplication is intentional, as it will enable the verification of information between these two groups of respondents, and it is hoped that it will be temporary if SBA finds consistency amongst respondents.

Cluster administrators interact with many of their participating small business on a near daily basis, and are expected to collect information about the progress of their participants as part of their contracts with SBA. Many also collect and use this information to better assist their participants and identify resources that are best suited to support their growth. They are therefore well positioned to obtain and report on this performance area. However, they do not necessarily collect this information systematically, nor do they interact with all their participating small businesses on a regular basis. As a result, they may not provide data that is as complete and accurate as that provided by small businesses. The goal is to better understand which group of respondents can most reliably provide this information. Information provided by these two groups will be cross-referenced and assessed for coverage based on each respondent group’s overall response rate and item nonresponse rates. Findings from this process will be used to identify which respondent group should be targeted to best obtain information in subsequent years, and any overlapping questions will be culled from one of the survey instruments. As part of this decision process, limiting the burden on small businesses will be considered such that, if the two sources of information are relatively similar, the cluster administrators will be targeted as a priority.

Consequences of Collecting the Information Less Frequently

Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The consequences of forgoing this data collection annually are twofold. First, without this data collection, SBA would not have systematic information on RIC participants’ post service responses to monitor trends and determine whether the program is meeting its goals and effecting change in the growth of target businesses or whether particular sites are in need of extra oversight and support from program managers at SBA. Information about participants’ satisfaction and level of involvement is also important to guide programmatic changes to the initiative by SBA, and adjustments by cluster administrators to the services they provide. Furthermore, the information collected is needed to produce OED’s annual report discussing its various initiatives. SBA’s other economic development programs and initiatives (e.g., SCORE and the Small Business Development Centers [SBDCs]) conduct similar client surveys annually, so it is important that RIC performance be represented to SBA and its stakeholders.

Special Circumstances Relating to the Guideline of 5 CFR 1320.5

Explain any special circumstances that would cause an information collection to be conducted in a manner:

requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than 3 years in connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or that unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary, trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

SBA requests that the data-submission period be fewer than 30 days (approximately 20 days). The shorter time frame, coupled with frequent notifications from multiple sources, would allow for the survey request to remain more prominent in clients’ minds. Shortening the response time could help increase the response rate by reducing the likelihood that a client would choose to procrastinate. No other special circumstances would cause information to be collected in the manners described above.

Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice, and describe actions taken by the agency in response to these comments. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.

The required public comment notice was published in the Federal Register on June 9, 2015, at 80 FR 32647 (see Appendix A-3). All comments were due on or before August 10, 2015. No comments on this collection were received.

Explanation of Any Payment or Gift to the Respondents

Explain any decision to provide any payment or gift to respondents, other than re-enumeration of contractors or grantees.

No payments or gifts will be made to respondents.

Assurance of Confidentiality Provided to Respondents

Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

In the study reports produced for the public, participant data will be reported in the aggregate, and information obtained from the interviews will be attributed to participants without any identifying information. Any personal information collected will be protected to the extent permitted by law, including the Privacy Act of 1974 and the Freedom of Information Act (FOIA). In addition, to the extent permitted under FOIA, in particular exemption 4, confidential business information will only be disclosed to contractor or Agency personnel assigned to administer the Initiative services and surveys.

Optimal, an independent contractor, will be collecting, compiling, and analyzing all data collected from RIC participants and RIC initiative administrators. The data will be collected through, stored, and analyzed on secure servers. Optimal implements administrative, physical, and technical controls to protect information collection. Secure intranets are used to maintain project-related files, and secure servers employ industry-standard methods, such as firewalls, monitored-access logs, virus protection, encrypted connections, password-protected accounts, and user-authentication mechanisms, to ensure the privacy of personal data. Optimal maintains a biometrically (physically) secure environment and employs a data security officer who oversees Optimal’s data. The security approaches are further described in Appendix A-2.

Justification for Sensitive Questions

Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

Cluster small businesses will be asked about their businesses’ financial situations, including revenue and external financing obtained. These measures are essential to the mission of the RIC Initiative and SBA to monitor business performance. As described in Part B, the information will be used to report on growth of participants, relative to a benchmark set of firms and relative to subsequent years of the initiative. Participants will also be asked for two unique business identification numbers, their DUNS and EIN numbers.

These data are necessary to improve data validity and reduce survey burden in future years of the study or in follow-up analysis on current participants. The DUNS will allow SBA to link RIC participants to the data it houses on loans and contracts. Additionally, SBA is pursuing partnerships to conduct program evaluations using federal statistical administrative data at the Internal Revenue Service and Census Bureau, which maintain economic data on businesses that is linkable via EIN. The potentially sensitive nature of these identifiers means the data will be transferred and maintained according to a data security protocol (see Appendix A-2).

Estimates of Hour Burden, Including Annualized Hourly Costs

Provide estimates of the hour burden of the collection of information. Indicate the affected public, number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.

SBA estimates that 388 hours are needed to conduct this study over the course of a year. The total annualized hour and costs burdens are detailed in the exhibit below.

The affected respondent types for this data collection include RIC administrators (i.e., the contract awardees), small business participants, and cluster stakeholders (i.e., large organization participants, such as universities and corporations). Preliminary pretests were conducted for all data-collection instruments to derive the burden estimates.

The cost burden for small businesses is based on an estimated median annual income of $66,500 for small business owners, which equates to an hourly rate of $25. See http://www.payscale.com/research/US/Job=Small_Business_Owner_%2F_Operator/Salary. The cost burden for large organization stakeholders, $44 per hour, is based on a weighted average of the estimated mean hourly wages for management and nonmanagement positions for universities, nonprofits, government agencies, large businesses, and business associations (these types of organizations are expected to compose the majority of cluster large organization stakeholders). See for example http://www.bls.gov/oes/current/naics4_813400.htm.

Exhibit 1. Study burden estimate

Burden	Number of organizations	Frequency of collection	Average minutes spent per instance	Organization burden (in hours)	Organization burden (in dollars)
Small Businesses
Initial contact	821	1	3	41.1	$ 1,024.20
1st reminder	739	1	2	24.6	$ 614.60
2nd reminder	666	1	2	22.2	$ 553.89
3rd reminder	600	1	2	20.0	$ 499.00
4th reminder	540	1	2	18.0	$ 449.10
Small business survey	305	1	20	101.7	$ 2,536.58
Total small business burden				227.6	$ 5,677.37
Large Organizations
Initial contact	386	1	3	19.3	$ 848.93
1st reminder	348	1	2	11.6	$ 510.24
2nd reminder	314	1	2	10.5	$ 460.39
3rd reminder	283	1	2	9.4	$ 414.93
4th reminder	269	1	2	9.0	$ 394.41
Large organization survey	121	1	10	20.2	$ 887.05
Total large organization burden				79.9	$ 3,515.95
Cluster Administrators
Contact and scheduling for interviews	11	2	10	3.7	*
Interviews (1 mid-term, 1 annual)	11	2	150	55.0	*
Cluster administrator survey	11	1	120	22.0	*
Total large organization burden				80.7	*

Total overall burden				388.2	$ 9,193.32

* There are no additional costs associated with the administrators, as they are performing these functions under an existing contract.

13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information.

There is no other annual cost burden to respondents or record keepers.

14. Annualized Costs to Federal Government

Provide estimates of annualized cost to the federal government. Provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.

The annualized cost to the federal government is $275,990 (see exhibit 2, below). This includes the costs associated with the contractor conducting the project and the salaries of the assigned SBA employees.

Exhibit 2. Estimated annualized cost to the federal government

Estimates	SBA COR	SBA program manager	Evaluation contractor	Total federal costs
Number of employees	1	1	NA	NA
Hours anticipated	80	80	NA	NA
Average hourly pay	$43.67	$51.60	NA	NA
Total cost	$3,494	$4,128	$268,368	$275,990

The cost of the Office of Entrepreneurial Development (OED) employee involved in administration of the survey and study (the SBA Contracting Officer’s Representative [COR]) is estimated at GS-13, Step 1, at $43.67 per hour based on 2,080 hours per year. OED anticipates that this person will work 80 hours per year for 1 year. The annual cost for this OED employee over the course of this study is $3,494. The cost of the OED employee involved in study oversight (the SBA program manager) is estimated at GS-14, Step 1, at $51.60 per hour based on 2,080 hours per year. OED anticipates that this person will work 80 hours per year for 1 year. The annual cost for this OED employee over the course of this study is $4,128, although this would not be costs added. The annual cost for both of these OED employees over the course of this study is $7,622. Federal employee pay rates are based on the General Schedule of the Office of Personnel Management for 2015 for the Washington, D.C., locality.²

15. Explanation for Program Changes or Adjustments

Explain the reasons for any program changes or adjustments reported in items 13 or 14 of OMB form 83-I.

There are no program changes or adjustments to report; this information collection has not been reported previously.

16. Plans for Tabulation and Publication and Project Time Schedule

For collections of information whose results are planned to be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

This study will use quantitative (i.e., descriptive and inferential statistical techniques) and qualitative methods for analyzing the data. The analysis is expected to include tables for the statistics and potentially graphs. The full report will include the appropriate associated statistics, such as the p-value for statistical significance at the 10 percent, 5 percent, and 1 percent levels. Other data gathered from the surveys will be presented in tables using descriptive statistics. Qualitative analysis will involve determining challenges, lessons, and the factors to which participants attribute their success. This information may be showcased in text boxes or integrated into the text. A final public report will be released approximately 6 months after the data collection is complete. A standardized schedule for the project is attached in Appendix A-1.

17. Reason(s) Display of OMB Expiration Date Is Inappropriate

If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

SBA plans to display the OMB expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

Explain each exception to the certification statement identified in item 19, “Certification Requirement for Paperwork Reduction Act” of OMB Form 83-I. If Agency is not requesting an exception, the standard statement should be used.

SBA is not requesting any exceptions to the certification.

PART B. Collections of Information Employing Statistical Methods

Universe of Sampling Respondent Selection

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, state and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The U.S. Small Business Administration (SBA) is interested in deriving estimates of satisfaction and outcomes at the level of each cluster. Due to the relatively small size of each cluster’s small business and large organization participation levels, a census survey of each group will be conducted.

A small business participant is defined as one that received services from its cluster in the past fiscal year and thus excludes “inactive” participants in the clusters. A large organization stakeholder in the cluster is defined by the administrator of the cluster but must have participated in some way in the cluster during the previous fiscal year.

Exhibit 1 details the expected small business and large organization survey population size and response rate for each site and overall. Exhibit 1 uses participant data from previous instances of survey implementation for the first seven clusters (which were already involved in the initiative) and uses the average number of participants across the original seven clusters to extrapolate the participant counts of the four clusters that are new to this initiative and will be surveyed for the first time under this request. The response rates used in exhibit 1 have been derived in a similar fashion (known figures from previous year for the first seven clusters and extrapolation for the remaining four), and the estimated number of surveys completed is the product of each cluster’s estimated number of participants and estimated response rate.

Exhibit 1. Expected small business and large organization survey population size and response rate, per site and overall

Cluster	Est. number of small business participants	Est. response rate for small businesses	Est. number of surveys completed by small businesses	Est. number of large organization participants	Est. response rate for large organizations	Est. number of surveys completed by large organizations
Cluster 1	68	40%	27	8	38%	3
Cluster 2	34	91%	32	19	90%	18
Cluster 3	38	55%	22	9	33%	3
Cluster 4	267	12%	31	152	9%	14
Cluster 5	34	62%	22	22	46%	11
Cluster 6	52	69%	36	26	77%	20
Cluster 7	28	82%	23	10	80%	8
Cluster 8	75	37%	28	35	31%	11
Cluster 9	75	37%	28	35	31%	11
Cluster 10	75	37%	28	35	31%	11
Cluster 11	75	37%	28	35	31%	11
Total	821	37%	305	386	31%	121

Exhibit 1 does not discuss another source of data utilized in this evaluation: the cluster administrators. They are asked to provide data about services provided via an annual survey, and are interviewed twice a year to obtain qualitative information about the evolution of their organization and operations, etc. Again, there is no sampling, and all eleven cluster administrators will be asked to participate. In past years, all of them have submitted a survey and participated in the two annual interviews, leading to a 100% response rate (see exhibit 2, below). This pattern is expected to hold moving forward.

Exhibit 2. Expected cluster administrators survey population size and response rate, per data collection instrument

Data collection instrument	Total number of cluster administrators	Est. response rate for cluster administrators	Est. number of responses
Cluster Administrator Survey	11	100%	11
Mid-term cluster administrator calls	11	100%	11
Annual cluster administrator calls	11	100%	11

Describe the Procedures for the Collection of Information including:

Statistical methodology for stratification and sample selection; estimation procedure; degree of accuracy needed for the purpose described in the justification; unusual problems requiring specialized sampling procedures;, and any use of periodic (less frequent than annual) data-collection cycles to reduce burden.

As noted in question 1, no sampling strategy is used in this data collection. Data will be collected from all Regional Innovation Cluster (RIC) Initiative participants. The estimates are intended to be generalizable to the 11 RIC sites, which were selected to fulfill predefined SBA contract criteria but will not necessarily be representative of other regions of the United States or other locations in which the initiative may be deployed in the future. If the response rate falls below the 80 percent threshold included in the Office of Management and Budget (OMB) guidance, nonresponse bias analysis will be conducted and reported using business information collected in the Cluster Administrator Survey, which serves as the survey frame.

One objective of the study is to provide statistically valid estimates of the incremental change in key business outcomes of participants, including revenue and employment growth, as compared to the benchmark sample. To measure these differences, inferential statistics, such as the Student-T test, are used to compare the average difference between the two groups. The study will use a standard threshold of a 5 percent significance level to reject the null hypothesis of no difference. The benchmarking data come from the following sources:

The Quarterly Census of Employment and Wages (from the Bureau of Labor Statistics), which provides data on the number of employees
The Dun and Bradstreet (D&B) Business Database, which provides data on both revenue and number of employees

These data sources vary with respect to the frequency with which they are updated, the time periods covered, types of respondents, geographic and industrial granularity, and units of observation. However, the maximum amount of data and the closest match in time period will be used for the dataset.

The analysis does not allow for causal claims regarding the relationship between the outcome estimates and the services provided through the Initiative.

Describe Methods to Maximize Response Rates and Methods to Deal with Issues of Nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

To maximize the survey response rate and minimize respondent burden, the following data-collection techniques will be used:

Surveys will be introduced to the cluster participants via an official introduction e-mail from their respective cluster administrators. This will help legitimize the survey and highlight its importance. The introduction outlines potential benefits to the business community resulting from the evaluation. After the introduction letters, Optimal will send an e-mail to cluster participants with the direct survey link embedded.
Optimal will also send up to four reminder e-mails over a period of approximately one month to encourage potential survey respondents to complete the survey before the deadline. The reminders will be sent at different times of the day to increase the probability of reaching respondents at an opportune time.
Optimal will work with the cluster administrators to replace e-mail addresses on their respective rosters that are invalid or generate a bounce. This will ensure that as much of the survey frame as possible is given the opportunity to respond to the survey.
The surveys are designed to use skip patterns and simple, often multiple-choice answer options to facilitate the ease with which they can be completed. Surveys can be halted and resumed by participants to provide greater flexibility, and participants are free to skip question(s) that they may not know how to answer or do not want to answer.
To further maximize the response rate, Optimal plans to provide cluster administrators with a PDF smart form of the survey instruments. These smart forms are electronic, fillable versions of the survey that can be used if participants encounter technical difficulty or do not feel comfortable submitting information online. The clusters can also use these smart forms at their events or in the context of one-on-one counseling to capture data from a greater share of the cluster participants.

Describe Any Tests of Procedures of Methods to Be Undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The small business and large organization surveys were pretested internally by four individuals to ensure reliability, minimize measurement error, and minimize respondent burden. Feedback from these pretests was also used to identify problematic, difficult, or time-consuming questions and to revise or clarify them. The surveys were also revised based on the feedback received from the cluster administrators. In addition, once the surveys have been implemented on the web and as PDF smart forms, they will be tested by internal research staff for usability and to identify any errors related to their transfer to different mediums.

Pretests on internal research staff were completed to provide a burden estimate for the final small business and large organization survey instruments. Staff were instructed to complete the survey in one sitting, at a time and place with minimal distractions. A pretest of the cluster administrator survey was not possible, as the ability to answer the questions is influenced by the service tracking system the cluster employs and/or the frequency and variation in services provided to each participant business over the year. Discussions with administrators in seven clusters indicate that it requires approximately two hours on average. The average burden for each instrument is detailed in exhibit 3.

Exhibit 3. Burden estimates of survey instruments

Instrument	Average burden in minutes
Large organization survey	10
Small business survey	20
Cluster administrator survey	120

Expert Contact Information

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The consultants used to design, collect, and/or analyze the information for the agency are from Optimal Solutions Group, LLC. For questions regarding the study or questionnaire design or statistical methodology, contact the appropriate staff member(s) via the information below:

Optimal Solutions Group, LLC

M Square Research Park

5825 University Research Court, Suite 2800

College Park, MD 20740-9998

Telephone: 301-918-7301

E-mail: [email protected]

Specialization	Name	Title	Organization	E-mail address
Program Evaluation	Brittany Borg	Program Analyst	Small Business Administration	[email protected]
Public policy	Jennifer Auer	Project and IDIQ Director	Optimal Solutions Group	[email protected]
Labor economist	Mark Turner	CEO	Optimal Solutions Group	[email protected]

1 The PDF smart form is a PDF version of the web survey that can be filled out by respondents using PDF reader software (such as Adobe Acrobat Reader). The content entered is then saved and can be extracted to generate structured data. In addition, the PDF smart form can also be printed and filled out by hand. However, it must then be scanned and sent back by the respondent to submit the information.

2 The general schedule is found at http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/15Tables/html/DCB.aspx.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Kelsye Turner
File Modified	0000-00-00
File Created	2021-01-24

RIC PRA Supporting Statement 4-26-16