WBC Part B COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS_10-9-19

1. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

A national survey of WBC clients (WBCs are currently operating in 46 states, Washington D.C. and Puerto Rico) will be conducted to assess the short- and long-term changes in behavior and performance after centers provide services. Surveys will be completed by a sample of clients who received services in the previous year, thereby providing a minimum 12- to 18-month lag to measure business changes.

A single-stage, stratified sampling design will be used to select WBC clients. The universe includes clients from 115 WBCs; each WBC is managed by a WBC administrator. First, each of the eligible WBCs (approximately 92 WBCs based on previous data collection efforts) will be sent a request and, of the 92 WBCs, approximately 95 percent (87) of WBCs are expected to provide a full list of clients who received counseling or training at least 1 year ago.¹ The WBCs that did not provide their clients lists were unable to; they used sign-in sheets for training which did not contain the required individual client contact information. It is expected that approximately 53,546 client names will be provided by the eligible and responding WBCs. Second, a stratified sample of 4,000 clients, drawn from the full list of clients, will be selected according to WBC and pre-venture and in-business status across the estimated 87 WBC administrators who are expected to provide data.²

The full list of WBCs that delivered counseling or training services and an estimate of the number of clients served were obtained from SBA’s Entrepreneurial Development Information Management System (EDMIS). The system contains information from all WBCs and associated clients. However, the number of training clients served by each WBC is only reported in aggregate per training session; clients can attend multiple trainings. Thus, these initial client counts from EDMIS do not provide an unduplicated count of clients served per WBC and will be refined using the data provided by the approximately 87 WBCs. For the purpose of deriving a conservative sample size estimate, it was assumed that the 26,773 counseling clients are equal to the number of unduplicated training clients, with no overlap between the services. Precise figures will be calculated once data are received from the approximately 87 WBC administrators.

Table 3. WBC Client Universe, Sample, and Expected Response Rates

Strata	Counseling Clients in 2017	Counseling+ Training Clients (approx.) in 2017	Sample3	Expected Response Rate	Expected Respondents
Pre-venture clients	26,773	53,546	2,000	50%	1,000
In-business clients			2,000	50%	1,000
Total	26,773	53,546	4,000	--	2,000

Previous OMB-approved WBC client surveys achieved response rates that varied each year. The response rate to the 2018 WBC Client Outcome Survey (which collected information about client services received in 2016) was 9 percent. A 50 percent response rate is estimated for this collection because, unlike the previous studies, a targeted sample of clients will be surveyed, and additional follow-up activities will be undertaken (see B3 below).A nonresponse bias analysis was performed for the 2018 WBC Client Outcome Survey. A summary of the analysis is included in Appendix B10. A significant contributing factor was that training clients were over-represented in the sample and were less likely than counseling clients to respond. These results informed the current approach. Changes have also been made to the data collection methodology to help improve the response rate.

B.2 Describe the procedures for the collection of information including:

B.2.a Statistical methodology for stratification and sample selection

A single-stage, stratified sampling strategy at the WBC-client level is used to minimize cost, enhance the response rate, and maximize precision. A full list of operational WBCs in each year is available from the EDMIS data. All eligible WBCs (approximately 92) will be selected, and an estimated 87 WBC administrators will provide administrative data necessary to administer the WBC Client Outcome Survey.

Across the 87 client lists provided by WBC administrators, 4,000 clients will be randomly selected by strata (pre-venture and in-business and WBC) based on a universe of 53,546 and surveyed. There will likely be some strata that are very small (only one or two clients) and others that are much larger. If we encounter many very small strata, we may consider collapsing some strata to ensure that there are enough clients in all strata prior to sample selection. Alternatively, we may define the strata according to just one variable (only WBC or only client type) and control for the other variable using systematic selection of clients within strata. These alternative approaches will help reduce standard errors in the survey estimates if the strata sizes are highly variable. This sample includes the 4,000 WBC clients in the primary sample and 500 WBC clients in the back-up sample, which will only be released if response counts are lower than expected. The back-up sample of 500 will be drawn at the same time as the initial sample. We will continuously monitor response counts during data collection and propose utilizing the back-up sample at the mid-point of data collection (week 4) if the response patterns suggest we will not meet the goal of 2000 completed cases. The sample size was calculated using (1) a 95 percent confidence level, (2) a 5 percent margin of error, (3) an approximated design effect of 1.25 to account for weighting adjustments due to nonresponse, and (4) an estimated nonresponse rate of 50 percent. The level of precision is adequate for the purpose of performance monitoring. Once client data are received from the 87 WBCs, these statistics will be recalculated based on the actual population size and distribution between pre-venture and in-business clients. Prior to sampling, the sample frame will be reviewed for duplicate cases; duplicate clients will be determined within each center using the client name, email address, and WBC name. Duplicates will be excluded from the final sample frame prior to sampling.

B.2.b Estimation procedure and degree of accuracy needed for the purpose described in the justification

The survey estimates, which are descriptive statistics, will be derived by weighting responses to account for nonresponse and the adjusted probability of selection into the sample. The data will be weighted to adjust for client and business characteristics that influence outcomes provided with the client frame data, such as industry, race and gender. These characteristics were used in the past and determined to be the most appropriate. The data will also be weighted to adjust for the actual proportions of pre-venture and in-business clients in the universe. Ultimately, the above methodology will allow adequate population estimates and adequate estimates within the designated strata. Our design should provide us with 95% confidence intervals of +/- 3 percentage points or better for binary outcomes overall and +/- 7 percentage points or better for binary outcomes for strata with 250 or more completes.

B.2.c Unusual problems requiring specialized sampling procedures

SBA does not anticipate unusual problems that require the usage of specialized sampling procedures.

B.2.d Any use of periodic (less frequent than annual) data collection cycles to reduce burden

SBA does not plan to use data collection cycles that are less frequent than annual. Data collection will be conducted annually.

B.2.e General data collection procedures

A new sample will be drawn in each survey cycle. WBC clients may be randomly selected. A stratified random sample based on the factors identified above (WBC and business status) is used to obtain the most representative data possible.

Below is the proposed schedule for this study:

Task: Study and Survey Design	Date
60-day notice in Federal Register	10/29/2018
Draft OMB Materials	11/16/2018
Final OMB Materials	2/8/2019
SBA sends OMB materials for approval	4/16/2019
30-day notice in Federal Register	4/16/2019
OMB notifies SBA of approval	9/27/2019
Task: Build Sample Frame
Letter and template to WBCs	10/1/2019
Reminder email to WBCs	10/8/2019
SBA sends 2M completed templates for sample frame	10/29/2019
Select sample	11/4/2019
Task: Data Collection
Introductory letter to sampled clients	11/6/2019
Email invitation with web link	11/11/2019
1st email prompt	11/18/2019
2nd email prompt	11/25/2019
3rd email prompt	12/2/2019
Start telephone prompts	12/9/2019
4th email prompt	12/16/2019
Last email prompt	12/23/2019
Data collection ends	12/31/2019
Task: Data Delivery and Report
Clean data and derive estimates	2/7/2020
Draft report	2/28/2020
Final report	3/23/2020
Dissemination of Final Report	4/13/2020

The process of collecting data will occur in the following fashion:

1. At the beginning of each survey cycle, SBA obtains a list from all eligible WBCs of all WBC clients who have received at least one training or counseling service at least 1 year ago. SBA may contact WBC administrators in an effort to obtain this list by email (Appendices B2, B3, and B4). The sample will be stratified by business status (pre-venture or in-business). A primary sample size of 4,000 will be randomly selected, including estimated sample sizes of 2,000 pre-venture WBC clients and 2,000 in-business WBC clients.

2. WBC clients selected for participation in the WBC Client Outcome Survey will be notified of the study by mail (Appendix B5) and will be emailed the information necessary to complete the web survey (Appendix B6). These letters contain official labels to verify the organization’s intent and professionalism.

3. If the WBC client does not complete the web survey within 1 week, WBC clients may be contacted throughout the duration of the data collection period (up to five times by email [Appendix B7] and once by telephone [Appendix B8]) as reminders and encouragement to complete the WBC Client Outcome Survey via the web. The respondent will be provided a toll-free number to contact the study team at any point during data collection. The telephone interviewer will read a prepared script to the WBC client and in addition to encouraging participation, will verify the email address to ensure the web survey is being delivered to the appropriate person. depending on the client’s preference and availability.

4. Throughout the duration of the data collection period, web-based responses will be stored electronically and downloaded to a secure folder on the contractor’s internal project directory

B.3 Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Overall response rate projections are presented above in section B.1. Achieving the specified response rates involves using procedures described below to secure participation once the potential respondents have been identified. The WBC Client Outcome Survey is designed to achieve the highest practical rates of response, commensurate with the importance of survey uses, time constraints, respondent burden, and data collection costs. Efforts to minimize cost include the convenience of responding on an online platform, the short length of the survey (6 minutes), easy-to-answer question formats (close-ended, multiple-choice questions), the limited number of sensitive questions, and the requests for approximations or estimates of business growth (thereby minimizing time respondents spend searching records for answers). In the event that an 80 percent response rate is not obtained, a nonresponse bias analysis will be conducted .Nonresponse bias will be conducted prior to estimates being released. It will be a part of the data cleaning, processing, and analysis. The nonresponse bias analysis will involve looking at relationships (correlations) between response status and key demographic, sampling frame, and outcome variables to see if there are any patterns that would potentially influence outcome estimates.

Below, we describe the recruitment procedures designed to maximize the number of WBC administrators who provide administrative data needed to develop the sample frame for the WBC Client Outcome Survey.

• The introductory letters and emails to the selected WBCs will be carefully developed to establish the legitimacy of the study and its connection to the larger SBA goal, to inform the W BC administrators that the final report will be provided to them, and to request the WBC client list. These procedures help increase participation by increasing the perceived benefits of the data collection.

• Emailed follow-up attempts will be made with all WBC administrators who do not submit the administrative data request by the due date cited in the introductory letter. The primary purpose of the contact attempts will be to urge WBC administrators to submit the information as soon as possible.

Below, we describe the recruitment procedures designed to maximize the number of WBC clients who complete the WBC Client Outcome Survey.

• The introductory letters inviting WBC clients to participate in the WBC Client Outcome Survey will be carefully developed to emphasize the importance of this study and how the information will help SBA improve the WBC program, as well as maximizing cooperation through advance notice. All letters will include official SBA letterhead.

• A toll-free number and email address for the study will be provided so respondents can receive assistance with the survey.

• Sampled WBC clients will have the option of completing the WBC Client Outcome Survey on the web or by telephone.

• A training for telephone interviewers will be conducted. The training, specific to this study, will include an overview of the project, a primer on interviewing practices and procedures, techniques for encouraging respondent candor, and a review of the survey questions in case the interviewer needs to conduct the survey over the telephone.

• A core set of interviewers who have proven their ability to obtain cooperation from a high proportion of sample members will be assigned to the study.

• Follow-up attempts will be emailed to all sampled WBC clients who do not complete the survey 1 week after the start of data collection. During the last 4 weeks of data collection, telephone follow-up attempts will be initiated. The primary purpose of the contact attempt will be to urge WBC clients to complete the survey. At that point, if the WBC client chooses to complete the survey, an interviewer will administer the full survey or any remaining parts of the survey over the telephone.

• Call-scheduling procedures that are designed to call numbers at different times of the day (between 8:00 a.m. and 9:00 p.m., local time for WBC clients) and different days of the week (Monday through Saturday) will be used to improve response rates.

B.4 Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The WBC Client Outcome Survey (English) was pretested with four respondents for understandability (confusing wording or layout, failure to comprehend the question, etc.) and length of time to answer. The WBC Client Outcome Survey was revised to incorporate pretest results. Revisions included adding definitions to some terms in the survey and modifying the wording of the introduction email for clarity purposes. Burden estimates did not change. The final version of the English survey is included in Appendix A1, and the Spanish version was revised accordingly (Appendix A2).At this time, there are no future plans for testing.

B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The Contractor, 2M Research, will conduct this study. See Table 4 for contact information.

Table 4. Study Team Contact Information

Name	Affiliation	Telephone Number	Email
Dena Moglia	Senior Management and Program Analyst, SBA	202-205-7034	[email protected]
Laura Maas	Program Manager, SBA	202-205-7341	[email protected]
Nicholas Beyler	Project Director, 2M Research Services	703-214-0931	[email protected]
Mary Ann Latter	Lead Project Manager, 2M Research Services	703-214-1200	[email protected]
Amy Wieczorek	Research Analyst, 2M Research Services	817-856-0866	[email protected]

1 The estimated number of eligible, participating WBCs is based on SBA’s prior experience with this survey. The number of eligible WBCs is less than the universe of WBCs because not all WBCs are eligible to participate in the survey. To be eligible a WBC must meet two criteria (1) in operation for at least 3 years prior to year of observation and (2) the WBC must still be in operation.

2 This number includes 4,000 WBC clients in the primary sample frame. An additional sample of 500 WBC clients will only be added if response counts are lower than expected.

3 This number includes 4,000 WBC clients in the primary sample frame. An additional sample of 500 WBC clients will only be added if response counts are lower than expected.