WBC supporting statement 3-30-15

Supporting Statement for Collecting Data on Women’s Business Center Program Participants

(OMB Control Number: XXX)

PART A. Justification

1. Circumstances Making the Collection of Information Necessary

Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

The Women’s Business Center (WBC) program was established by the Women’s Business Ownership Act of 1988. The program provides entrepreneurial development services, such as business training, counseling, mentoring, and referrals, for business owners and those considering starting a business through a network of approximately 100 centers. SBA is required to report including its own program performance for all its funded programs, including the WBC program. However, there is no data collection currently in place to systematically track WBC program outcomes such as client satisfaction, adoption of new business practices, or changes in business size or scope. WBC administrators are not required to systematically contact clients regarding the outcomes of their visit. The client data collected by SBA (i.e. OMB-approved Form 641) are obtained only from clients returning to the WBC for more than one visit. In addition, OMB-approved Form 888 collects aggregate information on client training sessions (e.g. total attendees) so there is no outcome information for clients that solely attend trainings provided by WBC. This data collection will fill the gap by administering an outcome survey to a random sample of WBC clients receiving counseling or training (see appendix I. Survey Instrument).

2. Purpose and Use of the Information Collected

Indicate how, by whom, how frequently, and for what purpose, the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

SBA will use the data from this information collection to monitor and report the performance of the WBC program to stakeholders (e.g. Congress, WBC leadership). The survey is not designed to make causal claims about the impact of WBC services on clients. However it is sufficiently designed to produce representative results of client attitudes and stated changes in business practices and performance after receiving one or more services from WBC. Moreover, it provides clients the opportunity to suggest improvements to the program. SBA will monitor these suggestions and performance outcomes over time and provide the information to the Association of Women’s Business Centers (AWBC) to limit their need for a duplicative survey.

This outcome information will allow administrators to improve the program. Moreover, it allows WBC program administrators and SBA to identify whether trends correspond to intentional changes in program strategy and implementation.

3. Use of Information Technology and Burden Reduction

Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden. Insert any applicable electronic web address.

SBA is committed to complying with the 2002 E-Government Act to promote the use of technology in order to lessen the burden of data collection on the public. WBC client data will be collected via a web-based survey to reduce client burden. The screenshots of the web surveys are presented in appendix I. Web-based Survey. They are a cost-efficient solution that provide critical information from respondents in real time and allow flexibility in the survey’s format, layout, and navigation. The web-based survey minimizes the respondents’ burden and improves data quality by providing skip patterns for questions that are not applicable, pop-up instructions to questions, an easy navigation, and the ability to stop and restart the survey at any time convenient to a respondent, unlike a telephone survey. The Web-based survey also allows respondents to submit the final survey instantaneously upon completion.

4. Efforts to Identify Duplication and Use of Similar Information

Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

Review of existing studies did not identify any duplication of efforts in this data collection. SBA’s current data collection efforts (i.e. Forms 641 and 888) do not provide systematic information on client outcomes and do not contain measures of service satisfaction. The AWBC has collected client data periodically (e.g. 2010), but to our knowledge has no future plans to conduct a client survey. Other identified data collection or analytic efforts of women-owned businesses include those conducted by the National Women’s Business Council using federal administrative data, the Aspen Institute’s MicroTracker survey, and the Kauffman Foundation’s Firm Survey. These sources cannot be used to monitor performance of the WBC program because they do not adequately represent WBC clients. For example, many of the 107 WBCs do not participate in the MicroTracker survey and therefore their clients are not surveyed, and many WBC clients are not in federal administrative datasets (e.g. Survey of Business Owners, Longitudinal Business Database) because they have not yet started a business.¹

5. Impacts on Small Businesses or Other Small Entities

If the collection of information impacts small business or other small entities, describe any methods used to minimize burden.

This data collection will obtain information from small and nascent businesses. To minimize the burden on respondents, efforts have been made to simplify and streamline the survey. The estimated burden for completing the survey is only 6 minutes per respondent, largely due to the focus on a small number of behavioral questions and business information that does not require the respondent to retrieve data from a secondary source such as a tax return. The survey will be disseminated via an online platform, which will allow small business owners to take the survey at a time that is convenient for them and only answer questions that pertain to their experiences. For example, nascent businesses skip questions that pertain to business performance outcomes. The data collection also uses a sample design to minimize the number of small businesses from whom information is requested.

6. Consequences of Collecting the Information Less Frequently

Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The consequences of forgoing this data collection are two-fold. First, SBA’s other resource partners (i.e. SCORE and the Small Business Development Centers) conduct similar client surveys annually, so it is important that WBC performance is equally represented to SBA stakeholders annually. In addition, without this data collection SBA would not have systematic information on WBC clients’ post-service responses to monitor annual trends and changes and determine if the program is meeting its goals and affecting change in women-owned businesses.

7. Special Circumstances Relating to the Guideline of 5 CFR 1320.5

Explain any special circumstances that would cause an information collecti­on to be con­ducted in a manner:

• requiring respondents to report information to the agency more often than quarterly;

• requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• requiring respondents to submit more than an original and two copies of any document;

• requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;

• in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

• requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• requiring respondents to submit proprietary, trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

SBA requests review of three special circumstances. First, SBA requests the use of slightly different industry categories than the OMB standards in order to improve data quality and lower respondent burden. The survey provides respondents with 8 rather than 20 industry categories to choose from. They are a mix of two-digit and four-digit NAICS codes that are most relevant to sole proprietorships and other micro businesses operated by WBC clients² (see question 16 in appendix I. Survey Instrument). The standard 20 categories do not provide an adequate description of the relevant industries contained in them, which makes it difficult for clients to correctly self-classify. Having clients select from the full list of hundreds of four- or six-digit categories would increase the respondent burden. The 8 relevant categories, plus an “other” category are requested to minimize measurement error and minimize burden.

Second, the race and ethnicity categories used in the survey largely reflect the OMB Standards for the Classification of Federal Data on Race and Ethnicity. However, significant measurement error has been found in the standard, two-question format on race and ethnicity.³ SBA hopes to improve self-categorization by WBC clients by combining the question on Hispanic ethnicity and the question on race. SBA requests that the race categories allow respondents to select of one or more of the following options: American Indian or Alaskan Native, Asian, Black or African American, Hispanic or Latino, Native Hawaiian or Other Pacific Islander, White, Other race (see question 28) rather than use the two-question format.

Third, SBA requests that the data submission period be less than 30 days. The shorter time frame allows for the survey request to remain more prominent in clients’ minds through frequent notification from multiple sources and increasing the immediacy of the survey can help increase the response rate by reducing the likelihood that a client will choose to procrastinate.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.

A notice of this data collection was published in the Federal Register on June 4, 2014, at 79 FR 32356 to solicit comments on the survey and burden as required by the regulations in 5 CFR 1320.5 (see appendix II. Notice 79 FR 32356). SBA received no comments from its solicitation.

SBA consulted with Optimal Solutions Group, LLC (Optimal) to design the methodological approach for data collection and the survey instrument. Optimal’s team is composed of researchers with experience in diverse disciplines from labor economics to entrepreneurship and program evaluation. SBA also consulted with program administrators at three WBCs to provide additional information on aspects of the collection such as its frequency, the clarity of instructions, and which data elements to record.

9. Explanation of Any Payment or Gift to the Respondents

Explain any decision to provide any payment or gift to respondents, other than re-enumeration of contractors or grantees.

No payment will be made to the respondents.

10. Assurance of Confidentiality Provided to Respondents

Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

An independent third-party contractor, Optimal, will be collecting, compiling, and analyzing all data collected on WBC clients. The survey invitation letter informs the respondent that the survey data will be held in confidence by SBA and Optimal (see appendix III. Introduction Letter). It states that no survey participants will be identified in any public reports, as the collection is purely for statistical reporting purposes. Assurance of confidentiality, to the extent permitted, is provided by the Privacy Act of 1974, 5 U.S.C. 552a and the Freedom of Information Act, 5 US.C. 552, exception (b)(4), which allows SBA to deny FOIA requests for information disclosures that could cause substantial competitive harm. Contractors are required to adhere to the Privacy Act, which is stated in the contract with Optimal.

Optimal implements data management security protocols to protect personal and sensitive information. Secure intranets are used to maintain project-related files, and secure servers use industry-standard methods such as firewalls, monitored access logs, virus protection, encrypted connections, password-protected accounts, and user authentication mechanisms to ensure the privacy of personal data. Optimal maintains a biometrically (physically) secure environment and employs a data security officer who oversees Optimal’s data. The security approach used to protect the restricted-use data (RUD) is based upon Defense in Depth principles. In general, the RUD enclave consists of a single web portal accessible only via a virtual private network (VPN) and remote desktop software. The server does not have access to the Internet, nor does it have public IP addresses open. Access is only granted via the secure VPN connection. The detailed description of data handling, storage and disposition, and the procedures for privacy safeguards and protection are presented in appendix IV. Data Security Protocols.

11. Justification for Sensitive Questions

Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

The data collection does not request information typically considered private and sensitive in nature. However, WBC clients will be asked about their businesses’ financial situation, such as total revenue and the value of loans obtained. Although potentially sensitive, these measures are essential to the mission of the WBC program and SBA to monitor business performance. A reminder of the survey’s confidentiality statement is provided directly prior to these financial questions. Clients will also be asked for their gender, race, ethnicity, veteran status, disability status, and income in order to determine whether or not the WBC is fulfilling its mission of serving low-income women and the extent to which SBA services are provided with equal opportunity across different demographic groups. To determine if clients are low-income, the income categories are specially grouped and, when combined with number of people in the household, can be used to determine poverty status. This is an additional measure taken to limit the sensitivity a client may have about her income status. If these demographic data are found to be complete in the administrative records, the questions will be dropped from the survey in subsequent years.

12. Estimates of Hour Burden Including Annualized Hourly Costs

Provide estimates of the hour burden of the collection of information. Indicate the affected public, number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.

The total number of burden hours is 1496. The estimate is made of up both administrative and respondent burden. The estimate is based on an anticipated 1,145 respondents and an average survey completion time of 6 minutes. This estimate was developed by assuming a 25 percent response rate from the sample of 4,578 clients and by testing the time to completion with nascent and in-business clients of the WBC. More information about these estimates is presented in part B of this package. The estimated annual costs burden for this data collection is $20,653 to clients. The cost estimate is based on the average survey hours and an average hourly wage of $16.85.⁴ The other components of the total hours burden is shown in the table below.

Estimates	Survey introduction email	Survey invitation email	Survey reminder emails	Survey completion (client burden)	Total burden
Number clients	25472	4578	4578	1145	NA
Frequency of collection	1	1	4	1	NA
Average minutes spent	1	1	2	6	NA
Average hourly pay	$ 16.85	$ 16.85	$ 16.85	$ 16.85	NA
Client burden (in hours)	425	76	610	114	1,226
Administrative burden WBCs (in hours)*	170		100		270
Total client burden (in $)	$ 7,153	$ 1,286	$ 10,285	$ 1,928	$ 20,653

*There are no additional costs associated with the WBC burden as they are performing it under an existing grant.

Table 1: Hour Burden Estimate

13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information.

N/A – there is no other cost burden to respondents or record keepers beyond the cost reported above.

14. Annualized Costs to Federal Government

Provide estimates of annualized cost to the Federal government. Provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.

The annualized cost to the federal government for data collection and reporting is $111,862. The figure includes costs associated with the contractor conducting the project, the assistance provided by WBC administrators, and the federal staff overseeing the project. Specifically, the cost of the contractor to complete the study design, instrument design, data collection, OMB supporting documentation, data analysis, and report writing is $97,800, as documented in contract number SBAHQ-13-D-0003. The annual cost of assistance provided by the 45 WBC administrators is $8,535. The estimate is based on an hourly wage of $31.61 and 6 hours of assistance per site to supply the client list, send an initial notification to those on the list, and conduct follow-up reminder e-mails or calls.⁵ The annual cost of federal employee oversight of the project is estimated to be $5,528. The estimate is based on one Research Analyst GS-11, step 1 with an hourly wage of $30.64 and 12 hours of administrative oversight. In addition, it is anticipated that one Senior Advisor, GS-14, step 1 with an hourly wage of $51.60 will spend 100 hours in oversight of the project.⁶

Estimates	WBC administrators	SBA COR	SBA Senior advisor	SBA contractor	Total federal costs
Number of employees	45	1	1	NA	NA
Hours anticipated	6	12	100	NA	NA
Average hourly pay	$ 31.61	$ 30.64	$ 51.60	NA	NA
Total cost	$ 8,535	$ 368	$ 5,160	$ 97,800	$ 111,862

Table 2: Annualized Cost to Federal Government

15. Explanation for Program Changes or Adjustments

Explain the reasons for any program changes or adjustments reported in items 13 or 14 of OMB form 83-I.

N/A - this data collection request is new.

16. Plans for Tabulation and Publication and Project Time Schedule

For collections of information whose results are planned to be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

This study will use quantitative methods to analyze the survey data and produce tabulations in a public report. The descriptive statistics will include frequencies, averages, and ranges for the data elements. Standard errors and 95 percent confidence intervals will be provided for the estimates in a technical appendix. The reported estimates will be weighted, and unweighted estimates will potentially be contained in the technical appendix. If the report includes cross-tabulations, inferential statistics will be used to determine whether or not the group differences are statistically significant at conventional levels. The report is expected to include tables for the descriptive statistics in the technical appendix and graphical elements in the body of the report, such as pie charts and maps in order to visually represent the data. The survey will be administered in July 2015. The topline results will be provided to the AWBC upon completion. The complete schedule for the project is attached in appendix V. Project Schedule.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

N/A – data collection instrument will display the OMB approval expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

Explain each exception to the certification statement identified in item 19, "Certification Requirement for Paperwork Reduction Act." of OMB Form 83-I. Part V "Certification Requirement for Paperwork Reduction Act." If Agency is not requesting an exception, the standard statement should be used.

N/A – the agency is not requesting an exception to the certification statement.

PART B. Collections of Information Employing Statistical Methods

1. Universe and Sampling Respondent Selection

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

A national survey of WBC clients will be conducted to assess the short- and long-term changes in behavior and performance after centers provide services. Surveys will be completed by a sample of clients who received services in calendar year 2012, thus providing a minimum 18-month lag to measure business changes. Because WBCs offer both training and counseling services, clients who received either service will be included, as will clients who receive service by any mode of service delivery (e.g., telephone or face-to-face).⁷ A multistage, stratified sampling design will be used. First, 45 of the 107 WBCs will be selected at random. Each selected center will provide a full list of clients who received counseling or training in calendar year 2012. Second, a stratified sample of 4,578 clients will be selected according to the pre-venture and in-business clients across the 45 centers.

Exhibit 1. 2012 Universe, sample, and expected response rates

Strata	Counseling clients	Counseling+ training clients (approx.)	Sample	Expected response rate	Expected respondents
Pre-venture clients	17909	35818	2292	25%	573
In-business clients	12449	24898	2286	25%	572
Total	30358	60716	4578	25%	1145

The full list of WBCs that delivered counseling or training services and an estimate of the number of clients served were obtained from SBA’s Entrepreneurial Development Information Management System (EDMIS). The system contains information from all 107 centers and associated clients during the time period. However, the number of training clients served by each center is only reported in aggregate per training session; clients can attend multiple trainings. Thus, these initial client counts from EDMIS do not provide an unduplicated count of clients served per center and will be refined using the data provided by the 45 centers. For the purpose of deriving a conservative sample size estimate, it was assumed that the 30,358 counseling clients are equal to the number of unduplicated training clients, with no overlap between the services, and that their proportions of pre-venture versus in-business clients mirror that of the counseling population exactly, as indicated in the above exhibit. Precise figures will be calculated once data are received from the 45 centers.

Previous OMB-approved WBC client surveys were used to estimate a 25 percent response rate. These previous SBA surveys achieved response rates of 18 percent in 2013, 15 percent in 2012, and 20 percent in 2011. A 25 percent response rate is estimated for this collection because, unlike the previous studies, a targeted sample of clients will be surveyed, WBC administrators are helping implement the survey and additional follow-up and incentive activities are being undertaken (see question 3 below).

2. Describe the procedures for the collection of information

Statistical methodology for stratification and sample selection, estimation procedure, degree of accuracy needed for the purpose described in the justification, unusual problems requiring specialized sampling procedures, and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

A two-stage, stratified sampling strategy is used to minimize cost, enhance the response rate, and maximize precision. Stage 1 of sampling is at the center level; stage 2 of sampling is at the client level. For the stage 1 sampling frame, a full list of WBCs that were operational in 2012 is available from the EDMIS data. About half (45) of the WBCs will be selected to administer the client survey. Limiting the number of centers is a cost-effective way to obtain an unduplicated list of clients and the appropriate universe for each stratum because such a count is not available from EDMIS. The probability of each center to be selected is random and unbiased, which can be viewed as a Bernoulli random variable with parameter 0.5. In this sense, we are able to calculate the standard deviation, , and estimate that with 45 centers, the sample mean will be within 10 percent of the population mean with approximately 90 percent confidence.

Across the 45 WBC client lists, 4,578 clients will be randomly selected by strata, based on a universe of 60,716, and surveyed. The required sample size was calculated using 1) a 95 percent confidence level, 2) a 5 percent margin of error, 3) an approximated design effect of 1.5 (due to clustering from the multistage sampling design), and 4) an estimated non-response rate of 75 percent. Precision will likely be reduced for business finance items due to high variations in responses. Still, the level of precision is adequate for the purpose of performance monitoring. Once client data are received from the 45 centers, these statistics will be recalculated based on the actual population size and distribution between in-business and pre-venture clients.

The survey estimates will be derived by weighting responses to account for non-response and the adjusted probability of selection into the sample. The data will be weighted to adjust for client and business characteristics that influence outcomes provided with the client frame data, such as industry and race. It will also be weighted to adjust for the actual proportions of pre-venture and in-business clients in the universe. Ultimately, the above methodology will allow adequate population estimates and adequate estimates within the designated strata. No analysis of subgroups or specific centers will be conducted.

3. Describe methods to maximize response rates and to deal with issues of non-response.

The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

To maximize the survey response rate and minimize respondent burden, the study will use the following recommended data collection techniques and procedures:

• SBA will send an official introduction letter to the selected WBC centers to establish the legitimacy of the study and its connection to the larger SBA goal, informing them that the final report will be provided to them, and to request that the full client list for the counseling and training services be sent to the SBA contractor, Optimal (see appendix VI. Instructions for Selected Centers). These procedures help increase participation by increasing the perceived benefits of the data collection.

• Selected WBCs will send a pre-notification e-mail to their full client list about the upcoming survey, explaining the purpose of the survey and the random selection process (see appendix VII. Pre-notification Email). A pre-notification helps increase the response rate by building the client’s expectation of the survey and reducing the chance that the survey will be overlooked or disregarded.

• An official introduction e-mail from SBA and WBC will accompany the survey link sent (via the survey system) to selected clients (see appendix III. Introduction Letter). The letter will describe potential benefits of the survey to the program and small businesses to highlight the importance of the study. This introduction letter will be from the specific WBC at which the client received the service and is a key component to increasing the response rate. It improves the respondent’s trust in the purpose and use of the survey and according to social exchange theory; it will motivate a client to provide a response in exchange for the free service they received from the center.

• Up to three e-mail reminders will be sent at various times and on various days of the week to increase the odds of the survey invitations being noticed and acknowledged. Selected WBCs will be asked to send a final personal e-mail or phone call to non-respondents to encourage participation. The administrator will confirm that the respondent received the survey link and will obtain a new e-mail address if necessary (see appendix III. Follow-up Script). In the event that there are more than 10 non-respondents in one center, the administrator will send an e-mail rather than make phone calls to lower his or her burden.

In addition to the above efforts to increase the survey response rate over previous years, the response rate should improve by minimizing the cost of participation. Efforts to minimize cost include--making it convenient to respond via an online platform, the short length of the survey (i.e., 6 minutes), easy-to-answer question formats (close-ended, multiple-choice questions), few sensitive questions, and requests for approximations or estimates of business growth (thereby minimizing time respondents spend searching records for answers).

In addition to the above efforts to increase the survey response rate, the response rate should improve over previous surveys because of a minimized client burden due to the following factors: the short length of the survey (i.e., 6 minutes), use of a majority of close-ended and multiple-choice questions, and requests for approximations or estimates of business growth (thereby minimizing time respondents spend searching records for answers).

4. Describe any tests of procedures or methods to be undertaken.

Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from ten or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The survey instrument was pretested and revised in two phases to ensure reliability, minimize measurement error and to estimate the respondent burden. First, it was administered to two WBC clients using a cognitive interviewing method. Respondents were asked to verbalize their thought process by “thinking out loud” with a researcher on the phone while the respondent completed the survey online. This approach utilized follow-up probes to determine whether respondents were having difficulty understanding the meaning of the questions or any words, problems remembering the information needed to answer questions, issues with unnecessary questions or response options, or trouble with missing questions or response categories. The items that were reported by the respondents or were noted by the researcher as problematic, difficult, or time-consuming to answer were further discussed to determine the reasons for the difficulties. The second revision was based on the feedback received from four invited client respondents. These pretests were done retrospectively so they could also serve as a burden estimate. Respondents were instructed to complete the survey in one sitting, at a time and place with minimal distractions. Burden estimates include the time it takes respondents to review instructions, search data sources if needed, and complete, review, and submit their responses. The average burden, across in-business and pre-venture clients is 6 minutes (see exhibit 2).

Exhibit 2. Pretest survey

Respondent	Time to completion
Client 1	4 minutes
Client 2	5 minutes
Client 3	7 minutes
Client 4	8 minutes
Average burden	6 minutes

After the four respondents completed the pretest survey, a follow-up phone call was conducted with a researcher. Similar to the in-process cognitive interviews, the follow-up interviews included probes to determine whether respondents were having difficulty understanding the meaning of the questions or words, problems remembering the information needed to answer questions, issues with unnecessary questions or response options, trouble with missing questions or response categories, or problems with any sensitive questions.

Based on the two cognitive interviews and four pretest results, the surveys were revised to minimize the respondents’ burden. The revisions involved rewording some questions to improve comprehension, adding skip questions so respondents did not have to answer questions that were not applicable, adding an open-ended question, and revising some of the response scales. The final version of the survey is in appendix I. Survey Instrument.

5. Expert contact information

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Expertise	Name	Position	Affiliation	E-mail
Labor economics, program evaluation	Mark Turner	President & CEO	Optimal Solutions Group, LLC	[email protected]
Entrepreneurs, program evaluation	Jennifer Auer	Project Director	Optimal Solutions Group, LLC	[email protected]
Statistics	Panpan Zhang	Statistician	George Washington University	[email protected]
Survey methodology	Laura Leete	Senior Research Associate	Optimal Solutions Group, LLC	[email protected]

For information on the statistical aspects of the survey design, contact the contractor implementing the study.

Jennifer Auer, Ph.D.

Project Director

Optimal Solutions Group, LLC

[email protected]

301-918-5086

1 A previous survey of WBC clients conducted by SBA showed that about 35 percent had not yet started a business (https://www.sba.gov/sites/default/files/files/OED_ImpactReport_09302013_Final.pdf).

2 A previous survey of WBC clients conducted by SBA showed that over half did not have any employees (https://www.sba.gov/sites/default/files/files/OED_ImpactReport_09302013_Final.pdf).

3 See http://www.pewsocialtrends.org/2012/08/07/census-bureau-considers-changing-its-racehispanic-questions/.

4 WBC clients’ hourly wages were estimated from the Current Population Survey, 2014 Annual Social and Economic Supplement, Table PINC-07. For self-employed women, the mean annual wage is $35,045 and the mean hourly wage, based on a 2,080-hour work year, is $16.85.

5 Administrators’ hourly wages were estimated from the Bureau of Labor Statistics Occupational Employment and Wages, May 2013. For 11-9151 Social and Community Service Managers, the mean hourly wage is $31.61 and the mean annual wage is $65,750.

6 Federal employee pay rates are based on the General Schedule of the Office of Personnel Management for 2015 for the Washington D.C. locality and a 2,080-hour work year.

7 Although the expected effect on business outcomes may be smaller when a client receives only a few hours of counseling or training, a survey of WBC centers found that the average client in over one-third of WBCs received less than 5 hours of training. Thus, to best represent WBC clientele, the sample will include clients who received any level of service (This information comes from a survey of WBC administrators sent by the association of WBCs on March 24, 2014. It obtained a 33 percent response rate as of April 11, 2014).

10