1205-0505 Supporting Statement_Part A_2016-02-08

Self-Employment Training (SET) Demonstration Evaluation

OMB Control # 1205-0505

December 2015

CONTENTS

1. Circumstances Necessitating the Data Collection 2

a. Policy Context 3

b. Prior Research on Self-Employment Services 4

c. The SET Program Model 6

d. Selecting Study Sites and Participants 8

e. Overview of SET Evaluation 9

2. How, by Whom, and for What Purpose the Information Is to Be Used 13

3. Uses of Technology for Data Collection and to Reduce Burden 17

4. Efforts to Identify Duplication 18

5. Methods to Minimize Burden on Small Businesses or Entities 18

6. Consequences of Not Collecting the Data 19

7. Special Data Collection Circumstances 19

8. Federal Register Notice and Consultations Outside of the Agency 20

a. Federal Register Notice and Comments 20

b. Consultations Outside of the Agency 20

9. Respondent Payments 20

10. Privacy 21

a. General Policies to Protect Privacy 21

b. Systems Security 22

c. Treatment of Data with Personal Identifying Information Collected for the Evaluation 23

11. Questions of a Sensitive Nature 25

12. Estimated Hour Burden of the Collection of Information 26

13. Estimated Total Annual Cost Burden to Respondents and Record Keepers 27

14. Estimated Annualized Cost to the Federal Government 27

15. Changes in Burden 27

16. Publication Plans and Project Schedule 27

17. Reasons for Not Displaying Expiration Date of OMB Approval 28

18. Exceptions to the Certification Statement 28

REFERENCES 29

TABLES

A.1 Research Questions for Implementation Study by Data Source 11

A.2 Research Questions for Impact Analysis by Data Source 13

A.3 Data Elements in the Follow-Up Survey, by Content Area and Availability of Similar Measures in Other SET Data Sources 15

A.4 Annual Burden Estimates for SET Demonstration Follow-up Survey, February 1, 2016 to September 30, 2017 26

A.5 Monetized Burden Hours for the SET Demonstration Follow-up Survey, February 1, 2016 to September 30, 2017 27

The U.S. Department of Labor (DOL), Employment and Training Administration (ETA) contracted with Mathematica Policy Research to implement and evaluate the Self-Employment Training (SET) Demonstration. This demonstration is a reemployment program targeted towards dislocated workers, as defined by the Workforce Investment Act (WIA)¹, who are interested in starting or growing a business in their fields of expertise. The demonstration is seeking to connect such workers to self-employment training, intensive business development assistance, and other services (including seed capital microgrants) to help them become more successful in self-employment.

The main objective of the evaluation of the SET Demonstration is to understand whether providing dislocated workers with access to intensive business development services and self-employment training increases their likelihood of reemployment, their earnings, and their propensity to start a business. The evaluation uses a rigorous experimental design in which approximately 3,000 applicants to the program in six study sites are randomly assigned to a program group or a control group with equal probability. Members of the program group receive ongoing access to intensive business development counseling from a self-employment advisor, as well as training and other assistance related to their specific self-employment needs, free of charge to them, for up to 12 months. Program group members who achieve key program participation milestones (such as completing a business plan) have the opportunity to apply for seed capital microgrants of up to $1,000 to help pay for inventory, equipment, licenses, or other business establishment costs. The control group does not have access to SET services during the implementation period and are ineligible for the SET microgrants. Both groups are able to seek out and make use of other self-employment services offered by existing community providers, although program group members will have such services partially subsidized through the demonstration. Impacts are being measured 18-months after randomization. An implementation study is also being conducted to provide information that will help ETA further refine the self-employment services made available to dislocated workers and other customers of the workforce system.

This will be the second clearance package submitted to the Office of Management and Budget (OMB) for this evaluation. An initial data collection package, approved by OMB in January 2013 (OMB Control Number 1205-0505, Information Collection Reference (ICR) Number 201209-1205-001), requested clearance for the application package, program participation records, a follow-up survey, the evaluation team’s site visit and case study protocols. This data collection was approved with an expiration date of January 31, 2016. In October 2014, a non-substantive change request (ICR Number 201408-1205-005) was approved to shorten the administration time of the follow-up survey from 60 minutes to 20 minutes to reduce respondent burden. The non-substantive change did not affect any data collection efforts other than the follow-up survey.

This new request is to extend OMB clearance of the 18-month follow-up survey to September 30, 2017. Given that study enrollment has proceeded more slowly than originally planned, an 18-month follow-up survey could be administered to only approximately 25 percent of the demonstration applicants by the current expiration date of January 31, 2016. Assuming an 80 percent response rate, this would result in approximately 600 respondents (= 3,000 respondents × 0.80 response rate × 0.25 of study participants). Extending the expiration date to September 30, 2017 will allow sufficient time to field the survey to all study applicants. There are no proposed changes to the survey instrument or the way it is administered. This request does not cover any of the other elements of the OMB-approved data collection; no extension is required for the consent and application forms, the program participation records, or the evaluation team’s site visit and case study protocols.

This package includes:

1. Appendix A: 18-Month Follow-Up Survey Instrument (previously approved under ICR Number 201408-1205-005)

2. Appendix B: 60-Day Federal Register Notice

1. Circumstances Necessitating the Data Collection

ETA seeks to implement and rigorously evaluate the effectiveness of innovative strategies for promoting reemployment based on the authority granted to DOL under Title I of WIA.² The SET Demonstration focuses specifically on self-employment as a reemployment strategy for dislocated workers. The demonstration is premised on the hypotheses that (1) self-employment could be a viable strategy for dislocated workers to become reemployed; (2) starting a small business is difficult, especially for individuals who lack business development expertise or access to start-up capital; and (3) dislocated workers might experience difficulties identifying and accessing training programs and other forms of assistance that could effectively prepare them for self-employment via the existing workforce infrastructure.

The SET Demonstration implements a new service delivery model that seeks to better connect dislocated workers to self-employment training and related assistance. This approach differs from previous large-scale demonstration programs, which have provided mixed evidence on the effectiveness of self-employment services on earnings and employment, because the SET Demonstration will (1) offer more intensive business development services than prior demonstrations have offered, (2) concentrate on individuals who have fairly limited traditional employment prospects but are well-positioned to benefit from self-employment counseling and training, and (3) make available seed capital microgrants to those individuals who engage strongly in the program and achieve key participation milestones to help them establish their businesses. The SET Evaluation will assess the effectiveness of the SET Demonstration model.

The remainder of this section provides additional information on the context and nature of the SET Evaluation in five subsections. The first subsection provides an overview of the Federal policy environment in which the demonstration is being implemented. The second discusses prior ETA research projects that have examined the effects of self-employment service provision. The third subsection describes the main features of the program model that is being implemented in the SET Demonstration. The fourth describes the criteria that is used to select sites implementing the demonstration and the target population of dislocated workers that the demonstration seeks to serve. The fifth subsection outlines the main features of the SET Evaluation.

a. Policy Context

Federal policymakers place a high priority on increasing entrepreneurship, self-employment, and business ownership, while recognizing the difficulties that individuals face when starting a business, such as limited business development expertise and access to start-up capital. Notably, the American Recovery and Reinvestment Act of 2009 (ARRA) included $730 million for the Small Business Administration (SBA) to increase small business lending opportunities. Likewise, the Small Business Jobs Act (P.L. 111-240), signed by President Obama on September 27, 2010, sought to promote small business job creation through a variety of mechanisms, including increasing access to capital and targeted counseling and technical assistance. More recently, the Middle Class Tax Relief and Job Creation Act of 2012 (P.L. 112-96) included provisions for an expansion of state Self-Employment Assistance (SEA) programs serving dislocated workers receiving unemployment insurance (UI) benefits.

The goal of the SET Demonstration is to rigorously test whether access to a self-employment assistance package that is centered around intensive business development counseling from a self-employment advisor (described later) increases the employment, earnings, and rates of self-employment among dislocated workers. The transition to self-employment could be particularly challenging for dislocated workers because their work experience is strongly tied to previous wage and salary employment. Although some skills and substantive knowledge can be transferrable to self-employment, starting and running a small business often calls for a broader set of managerial skills (such as knowledge of finance, bookkeeping, marketing, and human resources) than what is required in many wage or salary jobs. Frontline staff members of the Federal workforce system, such as at WIA American Job Centers (AJCs) and state employment offices, also have typically focused on traditional wage and salary employment when supporting their customers in their reemployment efforts. AJC staff tend to be familiar with traditional job-search strategies and job-matching assistance or avenues for pursuing occupation-specific training, and have less experience in linking customers to providers of self-employment assistance and training. Thus, the service model implemented in this demonstration could improve the ability of the existing workforce system to leverage available self-employment services for dislocated workers interested in pursuing self-employment as a reemployment strategy.

The program model implemented in this demonstration potentially augments SEA programs, which are currently being implemented in seven states and serve a small subset of dislocated workers who are likely to exhaust their UI benefits and are interested in self-employment. A temporary program created in response to the North American Free Trade Act, SEA programs received permanent authorization in 1998. The seven states currently implementing a permanent SEA program are Delaware, Mississippi, New Hampshire, New York, Oregon, Rhode Island, and Vermont. Other states have passed enabling legislation, but their SEA programs are currently inactive. Eligibility for SEA programs is limited to UI recipients identified by state Worker Profiling and Reemployment Services (WPRS) systems as being likely to exhaust their regular UI benefits.

SEA programs seek to address the challenges facing such dislocated workers by providing self-employment training and technical assistance.³ However, although SEA legislation requires that participants be provided with such services, a comparison of programs found that “the intensity or extent of the provision varie[d] greatly” across SEA states (Kosanovich and Fleck 2002; page 11). The new SEA provisions contained in the Middle Class Tax Relief and Job Creation Act allow states the option of establishing SEA programs for individuals eligible for Extended Benefits (EB) or Emergency Unemployment Compensation (EUC). The new provisions also authorize the distribution of $35 million in grants to help states implement or improve administration of SEA programs (for both claimants of regular UI and extended benefits). These funds may be used to promote the SEA programs and enroll individuals in these programs, but cannot be used to pay for training.

The SET Demonstration could provide valuable information to states seeking to establish a more intensive service model to complement their SEA programs. The evaluation will provide evidence on innovative approaches that could be used to more consistently connect dislocated workers to services that may ease the challenges they face in their transition to self-employment. Additionally, the SET Demonstration is examining a broader dislocated worker population beyond likely UI exhaustees.

b. Prior Research on Self-Employment Services

ETA has examined how entrepreneurial services affect the outcomes of individuals seeking self-employment through two waves of prior research. During the late 1980s and early 1990s, ETA funded the implementation and evaluation of two UI Self-Employment Demonstration (UISED) projects. During the first decade of the 2000s, ETA sponsored and evaluated a multistate initiative known as the Growing America Through Entrepreneurship project (Project GATE). Prominent features and results of the UISED projects and Project GATE are described in additional detail in this subsection.⁴

UI Self-Employment Demonstrations. UISED programs were implemented in two states: (1) the Enterprise Project in Massachusetts and (2) the Self-Employment and Enterprise Development (SEED) Project in Washington state. In both sites, UI recipients interested in self-employment were offered the opportunity to attend self-employment classes and workshops that covered a core set of business training modules after attending an orientation session and completing an application to the program. Participants were also given financial assistance similar to participants in state SEA programs. There were two notable differences between the Massachusetts and Washington UISED programs. First, the Enterprise Project restricted eligibility to UI recipients who, according to the state’s Worker Profiling and Reemployment Services (WPRS) statistical model, were likely to exhaust their benefits, whereas the SEED project permitted all UI recipients to participate. Second, Washington allowed program participants who had completed certain required training activities to receive their remaining UI benefits in a lump sum that could be used as business seed capital.⁵ The Enterprise Project in Massachusetts did not include this provision.

Both UISED programs were evaluated using experimental designs that compared program and control group outcomes at 21 months and 32 months, on average, after random assignment. While the findings from both evaluations were generally positive, there were notable differences in the results between the states (Benus et al. 1995). In Massachusetts, the Enterprise Program increased the propensity to enter self-employment during the evaluation period by 12 percentage points—a substantial effect compared with the 50 percent rate of entry into self-employment seen in the control group. However, at the time of the final follow-up, there was no significant difference between the two groups in the percentage of individuals who remained self-employed. The Enterprise Project also increased overall earnings during the follow-up period by more than 50 percent, but the higher earnings were largely due to wage and salary jobs, not self-employment. In Washington, the rate of entry into self-employment was 22 percentage points higher in the SEED program group than in the control group—63 percent, compared with 41 percent—by the last follow-up. The SEED program also substantially increased the likelihood of remaining in self-employment through the end of the study period by 12 percentage points. Yet, the program had no effect on employment or total earnings at any point, because the increase in the rate of, and greater earnings from, self-employment were almost exactly offset by a decrease in the rate of, and earnings from, wage and salary employment.

Project GATE. Although the previous Federal demonstration projects had targeted UI recipients exclusively, Project GATE sought to use AJCs to help a broader population of interested participants gain access to self-employment training and technical assistance, including help in applying for business loans. The AJCs conducted outreach and hosted the program’s orientations. Although many of the services available to Project GATE participants were already available in the community from Small Business Development Centers (SBDCs), Women’s Business Centers (WBCs), and community-based organizations (CBOs), the demonstration project sought to augment this existing infrastructure in three important ways:

1. Any individual who expressed an interest in self-employment and completed an application was enrolled in the demonstration.

2. Program participants received a formal assessment after being enrolled in the program to establish services most appropriate for them. After this assessment, participants were referred to a single provider (that is, an SBDC, a WBC, or a CBO) to attend entrepreneurship classes and/or receive technical assistance.

3. All services were provided to Project GATE participants free of charge.

Project GATE was implemented in five sites—Philadelphia, Pittsburgh, Minneapolis-St. Paul, Northeast Minnesota, and the state of Maine—from fall 2003 to summer 2005.

Project GATE was evaluated using a random assignment design in which the program group, which could receive demonstration services, was compared with a control group that could not participate in Project GATE, but could access any other self-employment services available in its communities. The evaluation found that the program had a significant, but small, impact on the rate of business ownership in the early quarters after program enrollment, but this impact eroded over time (Benus et al. 2008). By the 18-month follow-up survey, members of the program group were 3 percentage points more likely to own a business, relative to the control group (in which the rate of business ownership was 40 percent). The impact of GATE was slightly higher on individuals receiving UI benefits at the time of random assignment at 18 months: 5 percentage points, compared with an insignificant 2 percentage point differential between UI nonrecipients in the program and control groups. However, five years after random assignment, Project GATE participants and control group members were equally likely to own a business (Benus et al. 2009). Further, the five-year follow-up also indicated that Project GATE had no significant impact on total employment at any point during the five years after randomization and no impacts on UI benefit receipt, receipt of public assistance benefits, or household income. Finally, although the GATE program group earned less than the control group during the first six months after random assignment—presumably because they were engaged in entrepreneurial training—there was no statistically significant effect of GATE on total earnings at any other time during the remainder of the five-year evaluation period.

c. The SET Program Model

The SET Demonstration builds on the lessons from previous demonstration projects by implementing an intensive business development counseling model that (1) promptly engages program group members in services, (2) periodically (re)assesses their evolving needs and helps link them to the most appropriate services, and (3) provides ongoing motivation and support to overcome obstacles and persist in efforts at self-employment (as appropriate). As described in the next subsection, the demonstration services are targeted toward dislocated workers whose prior experience makes them likely to benefit from self-employment assistance and training. Another important component of the demonstration is the offer of seed capital microgrants of up to $1,000 to those program participants who engage strongly with the program and achieve key participation milestones (such as completing a business plan), to help them cover important business establishment costs.

A self-employment advisor at a local partner organization is assigned to each member of the SET program group. Similar to the GATE demonstration, this advisor assesses the participant’s initial situation, develops a customized package of training and technical assistance services tailored to the specific needs of the participant, and helps connect SET participants with appropriate providers in the community. The demonstration’s program model also calls for the self-employment advisor to promptly engage SET program group members (within a week of random assignment) and develop a deeper ongoing relationship with these clients, by meeting periodically to assess the participants’ progress toward self-employment and their evolving service needs, and by facilitating linkages to appropriate self-employment assistance services. Although the package of specific self-employment supports is customized for each participant, a common theme among self-employment advisors is to promote business development outcomes that are associated with self-employment success. Examples of such intermediate milestones include gaining access to startup capital, registering a business, and completing a business and/or marketing plan, as applicable.

SET program group members vary substantially in their self-employment assistance needs. Knowledgeable and experienced self-employment advisors are ideally positioned to help these participants identify and marshal the most appropriate and effective resources that are readily available in their communities. Recommended services may include microenterprise training, individualized business development counseling and technical assistance, access to mentors or peer support/networking groups, and other logistical support (such as incubator office space, discounted business services, meeting rooms, or high-speed Internet access). Hence, the demonstration’s approach aims to provide greater flexibility in developing a customized package of services for SET program group members than was feasible under Project GATE, which relied on a relatively small number of pre-specified providers and offered a core set of services. Preliminary research by the SET evaluation contractor suggests that limited staff availability and other resource constraints prevent many microenterprise development organizations (MDOs) from widely adopting an intensive business development counseling model similar to that proposed for the SET Demonstration. Thus, this approach has the potential to generate substantial differences between program and control group members in both the quantity and quality of services received.

Another important feature of the SET Demonstration is that program group members will be promptly engaged by their designated self-employment advisors after being accepted into the program. In Project GATE, initial assessments occurred, on average, 3.6 weeks after random assignment (Bellotti et al. 2006). This delay could have contributed to the 10 percent drop-off in the number of individuals receiving assessments, compared with the number of individuals who were randomized to the GATE program group. In light of these issues, the study team links SET program group members to partner providers with adequate capacity to engage them in services promptly after random assignment.

The SET self-employment advisors are also expected to follow up with program participants on a regular basis during a 12-month period to monitor their progress toward self-employment and periodically reassess their service needs. Such longer-term support is likely to represent an important improvement over the one-time, initial assessments used in Project GATE. As the needs of SET program participants change, self-employment advisors should be able to link them to additional self-employment services, if necessary, and help them troubleshoot difficulties that they encounter. Long-term follow-up is also expected to address the marked drop-off in service utilization over time seen in the GATE program, by promoting stronger engagement and persistence with self-employment efforts beyond the initial enrollment period.

A key component of the SET demonstration is the offer of seed capital microgrants to qualifying program participants. That is, those SET participants who engage strongly with the program and achieve key participation milestones have the opportunity to apply for and receive seed capital microgrants of up to $1,000 to help them cover direct costs associated with establishing and beginning to operate their businesses. Access to seed capital may be an important determinant of self-employment success for aspiring business owners, but few MDOs offer such support currently. Startup microgrants may help meet SET program participants’ immediate business capital needs while they wait for microloan applications or other funding sources to be approved. Such microgrants could help pay for start-up equipment and supplies or help defray business costs that may otherwise prevent aspiring business owners from establishing their businesses. They may also improve access to other sources of seed capital among dislocated workers who have a poor credit history and limited collateral.

d. Selecting Study Sites and Participants

The SET Demonstration is being implemented in six study sites at which recruitment targets dislocated workers likely to meet the study’s eligibility criteria (described below). It is expected that up to 4,000 individuals will apply to the SET program after completing a web-based orientation session describing the program. The evaluation team will select 3,000 individuals meeting the eligibility criteria—the application process will be closed once this target is reached but no later than January 31, 2016.⁶ Successful applicants are randomly assigned to a program group and will have access to the services described previously, or to a control group, who will not be entitled to receive the demonstration services.

Selecting a study population likely to benefit from SET services. The services offered as part of the SET Demonstration are concentrated on dislocated workers who, at baseline, already have established behaviors suggesting that they will be responsive to and benefit from self-employment training. In Project GATE, any interested individual could enter into the program, and the UISED projects enrolled all UI recipients (or likely exhaustees) who expressed an interest in self-employment. The SET Demonstration departs from this approach in recognition that self-employment might not be a realistic option for underprepared individuals. Moreover, concentrating scarce program resources on those individuals who are likely to receive some benefit from the program increases the likelihood that the SET Evaluation will detect statistically significant impacts. It also helps to avoid creating false expectations for success in self-employment among individuals likely to experience significant difficulties.

To identify dislocated workers who are likely to benefit from the program, the SET Demonstration concentrates on applicants whose prior work experience explicitly relates to their proposed business idea. This decision is motivated, in part, by research suggesting that individuals who have prior work experience in a related occupation, business, or industry are more likely to succeed as small business owners (Fairlie and Robb 2008; Harada 2003; Baptista et al. 2007). In addition, related work experience can be an important factor in lenders’ decisions to approve business loans. For example, the SBA’s web site notes that when lending institutions assess an applicant, “experience in business as well as past achievements in your industry will be reviewed.”⁷

To operationalize this targeting strategy, SET program applicants are asked to describe in detail how their proposed business idea relates to their prior work experiences. The eligibility criteria for the demonstration are explicitly described in SET publicity materials and during online orientations that all individuals interested in applying to the program are required to complete. At these orientations, prospective applicants are also informed that (1) applications not meeting SET’s eligibility criteria will be screened out and (2) meeting the eligibility criteria qualifies them only for a 50 percent chance to enter the SET program, based on the outcome of the random assignment lottery. Although eligibility criteria will also be made clear to interested individuals through mandatory online orientations, it is assumed that up to one in four applicants could be screened out after submitting an application.

e. Overview of SET Evaluation

The SET Evaluation will analyze the effectiveness of the SET Demonstration and will include two major components: (1) an implementation study and (2) an impact analysis. The results of the evaluation will provide ETA with valuable information to determine the extent to which the SET program model can serve as an effective and realistic approach for helping the target population of dislocated workers become reemployed.

Implementation study. This component of the evaluation will describe: (1) the implementation of the SET Demonstration program in each of the study sites and (2) the experiences of up to 32 individuals participating in the program. Much of the analysis for the implementation study is qualitative and is structured to address the following research questions:

• What is the context in which the SET Demonstration is implemented? Documenting the community setting and existing program infrastructure is essential for understanding the potential effects of the services provided as part of the SET Demonstration. This information will also enable the evaluation team to understand the “counterfactual” against which the SET Demonstration is being tested—that is, the entrepreneurial infrastructure that would be available to program participants in the absence of the intervention.

• What organizations participate in SET service delivery, and what are their responsibilities? Describing the characteristics and roles of workforce development partners and MDO providers is important for understanding the quality of program implementation and determining the kind of partnerships necessary for successful provision of SET services.

• What services are offered to program group members as part of the SET Demonstration and what other services are available to them? Examining the types of services offered through the SET program and how these differ from or complement microenterprise development and other services available to program participants will make it possible to determine whether the demonstration improves access to services. Of particular interest is whether SET providers are able to conduct ongoing follow-up with program group members, offer more customized technical assistance, and facilitate or provide increased access to capital.

• How well was the SET program implemented, and how did implementation vary across sites? Understanding any challenges in implementing the SET program model will provide ETA with information on the feasibility of scaling up the demonstration and/or lessons for how access to self-employment services might be improved in the future within the context of the Federal workforce system. Variation across sites provides information on contextual factors that might influence implementation. This variation can also help understand differences in the demonstration’s impacts, as discussed in later sections.

• What are the characteristics of the SET Demonstration study population? Learning about the baseline demographic and economic characteristics of applicants who met the study’s eligibility screens will help us understand which segments of the dislocated worker population were enrolled in the demonstration. This could provide ETA with insight on the extent to which the demonstration’s impacts have applicability in the broader customer base of the WIA-funded AJC system. In addition, comparing the characteristics of program participants with control group members will be essential for confirming the validity of the evaluation’s random assignment procedures.

• What were the experiences of selected SET Demonstration program group members? This component of the implementation study will seek to understand the experiences of SET program participants and the responses of selected participants to the SET program model. Of particular interest is whether SET provides customized guidance and intensive follow-up and whether the program helps address perceived barriers to starting a business, such as difficulties accessing startup capital and/or lack of technical expertise. In order to make efficient use of scarce resources, the study team will seek to gather additional detail about the experiences of both program participants who succeed in achieving key self-employment milestones and/or becoming self-employed and program participants who fail to do so. Examining differences in program experiences among participants who persist in their self-employment efforts and successfully establish a business, as well as those who do not and decide to focus on wage and salary employment, can shed light on the conditions and ways in which the demonstration’s services meet or fail to meet participants’ needs. Participants’ experiences can also suggest potential gaps in the overall service delivery infrastructure available to aspiring or nascent business owners, including SET demonstration services. For those who chose to focus on wage and salary employment, it will be useful to understand the role that SET had, if any, their decision to return to wage/salary employment (or to combine self-employment and wage/salary employment) and on the industry or occupation of the reemployed participant.

As shown in Table A.1, the implementation study relies primarily on site visits and case study interviews to obtain the data needed to answer these research questions, with additional information coming from the baseline SET Demonstration’s application forms, program participation records, and the follow-up survey.

Table A.1. Research Questions for Implementation Study by Data Source

	Data Source
Research Question	Application Package	Program Participation Records	Follow-Up Survey	Site Visit Interviews	Case Study Interviews
1. What is the context in which the SET Demonstration is implemented?				X
2. What organizations participate in SET service delivery and what are their responsibilities?				X
3. What services are offered to SET program group members and what are the other services available to them?		X		X
4. How well was the program implemented and how did implementation vary across sites?		X		X
5. What are the characteristics of the SET Demonstration study population?	X
6. What were the experiences of SET participants with the program?	X	X	X	X	X

Impact analysis. To rigorously estimate impacts, a randomized design is being used to compare the outcomes of approximately 1,500 program group members with the outcomes of approximately 1,500 control group members. Random assignment will enable the evaluation to obtain causal evidence on the effects of SET Demonstration services, relative to what might be obtained by members of the target population from existing community providers only. Additional information on the statistical methods used to estimate impacts and assess their statistical significance is presented in Part B of this package.

The impact analysis addresses the following research questions:

• What is the net impact of the SET Demonstration program on participants’ overall employment status and total earnings? In light of the goals of ETA for the SET Demonstration, as well as findings from the UISED demonstrations, two of the three primary outcome measures for the evaluation will be employment in any kind of job and total earnings. Findings on these outcomes will be used to summarize the overall effectiveness of the program in achieving its goal of improving the reemployment prospects of the dislocated workers served by the demonstration. The evaluation will also consider outcomes related specifically to self-employment and wage or salary employment separately. To augment the analysis of earnings, the analysis will examine job quality measures such as fringe benefits and availability of health insurance.

• Does the SET Demonstration increase the likelihood of self-employment? The evaluation will also consider the effects of the demonstration on participants’ likelihood of becoming self-employed and their likelihood of remaining self-employed through the end of the evaluation’s follow-up period. The latter measure is the third primary study outcome.

• Does the SET program improve intermediate business development outcomes? In order to better understand the channels through which the SET program operates, the evaluation will consider how effectively it encourages participants to take steps associated with self-employment success. The study will specifically consider intermediate milestones such as whether participants were able to gain access to startup capital, register their businesses, and develop and complete a business and/or marketing plan.

• How does participation in the SET Demonstration affect economic well-being and participation in other programs? It is also of interest to know the demonstration’s impacts on the economic circumstances of program participants. Accordingly, the impact analysis will examine total household income, measures of financial hardship, receipt of UI benefits, receipt of other forms of public assistance (including assistance from the Temporary Assistance for Needy Families [TANF] program and the Supplemental Nutrition Assistance Program [SNAP]), and receipt of government-sponsored job training and supportive services.

• Do program impacts differ for subgroups of participants defined by baseline characteristics? In addition to assessing whether the SET Demonstration worked, the evaluation seeks to shed light on the groups of individuals for whom the program has the greatest impacts. Accordingly, subgroup analyses will be conducted based on characteristics such as age, gender, race/ethnicity, education level, and previous work experience, industry, and occupation. Subgroups will also be formed based on psychological traits, such as risk tolerance, openness to new experiences, and perceptions of autonomy, which have been associated with entrepreneurial success by previous research (Calliendo et al. 2010, 2011; Evans and Leighton 1989). Receipt of UI benefits through state UI systems, the EB program, and/or the EUC program are of particular interest because the GATE evaluation provided evidence suggesting that UI recipients might have received early benefits from participating in GATE. Differentiating among UI recipients according to factors associated with their likelihood of exhaustion could also inform states’ use of WPRS models to identify candidates for SEA programs.

• Through what programmatic mechanisms might the SET Demonstration affect participant outcomes? It is important to know the extent to which individuals actually receive intensive business development counseling from a self-employment advisor, because this represents the key channel through which all other study outcomes are hypothesized to be affected. The analysis of this question will also examine the effects of participating in the SET Demonstration on the receipt of services at existing community providers. And, drawing on the results of the implementation study, the study team will determine whether impacts vary for participants in states and sites with different contextual or programmatic features.

The data used to answer these questions will primarily be derived from the demonstration’s application materials, program participation records, and follow-up survey. Additional information will come from the site visits and case study interviews conducted for the implementation study. Table A.2 displays how these various data sources map to the research questions for the impact analysis.

Table A.2. Research Questions for Impact Analysis by Data Source

	Data Source
Research Question	Application Package	Program Participation Records	Follow-Up Survey	Site Visit Interviews	Case Study Interviews
1. What is the net impact of the SET Demonstration program on participants’ overall employment status and total earnings?			X
2. Does the SET Demonstration increase the likelihood of self-employment?			X
3. Does the SET program improve intermediate business development outcomes?		X	X
4. How does participation in the SET Demonstration affect economic well-being and participation in other programs?	X		X
5. Do program impacts differ for subgroups of participants defined by baseline characteristics?	X		X
6. Through what programmatic mechanisms might the SET Demonstration’s program influence participant outcomes?			X	X	X

2. How, by Whom, and for What Purpose the Information Is to Be Used

Clearance is being requested for an extension to continue administering a follow-up survey. The extension will allow additional time to locate sample members and will lead to a higher response rate.

For both program and control group members, the follow-up survey (Appendix A) provides data on the outcomes of recipients 18 months after random assignment. An advance letter is mailed to study members shortly before fielding of the survey begins and provides information about the content of the follow-up survey, average administration time, and how to access the web-based instrument. Based on prior experience conducting similar surveys, the response rate is expected to be about 70 – 80 percent (that is, approximately 2,100 – 2,400 SET study participants are expected to complete the follow-up survey).

Descriptions of the major content areas of the SET follow-up survey follow. Unless otherwise noted, the information collected in the follow-up survey will be collected for use by the study team as an outcome in the impact analysis. Table A.3 describes the data elements included in the follow-up survey and how they relate to information available from the application package and program participation records. This extension request proposes no changes to the data collection instrument.

Screener. Before starting the survey, a screening section seeks to verify each respondent’s date of birth and the last four digits of the SSN. This ensures that the follow-up survey is only completed by individuals who went through random assignment.

Section A: Current employment status. At the start of the survey, respondents are asked whether they are currently self-employed and/or employed in a wage or salary job. Individuals with both forms of employment are asked to indicate which form of employment they consider to be their primary work activity. Individuals who are not employed are asked to describe any other recent work-related activities. This information will be used to construct a measure of respondents’ employment status at the time of the survey, which is one of the study’s primary outcomes, as well as a more general measure of labor force participation status that can be compared with existing data from the Current Population Survey.

Section B: Receipt of self-employment services. This section asks about self-employment assistance services accessed since random assignment. Sample members answer a series of questions about their receipt of such services as intensive business development counseling from a self-employment advisor, entrepreneurial classes, one-on-one technical assistance, peer support, mentoring, and assistance accessing loans. The survey also asks questions that quantify the amount of these services each sample member received and identify the organization that provided key services. Sample members who did not access any self-employment services are asked to specify their reasons for not doing so, which supplements the information collected in the implementation study on the appropriateness of the SET program model. Finally, respondents are asked about the extent to which the services they received addressed specific topics that are potentially important for aspiring small business owners, including achieving milestones (completing a business plan and obtaining start-up financing) that are expected to be associated with entrepreneurial success.

Section C: Self-employment experiences. This section of the survey seeks information about the sample member’s self-employment and business development experiences since random assignment. It includes questions about the number of self-employment ventures attempted since applying to the SET Demonstration and earnings from self-employment over the previous year. For individuals who started a business or other self-employment venture, a series of detailed questions is asked about the most recent business or venture, including the industry of the business, the period of operation, and the formal registration of the business. Respondents will also be asked about the hours they devoted to that business and the earnings they received as the owner. Each of these outcomes is interesting in its own right, and they may be used together to form a measure of the owner’s returns from business ownership. A measure of business profitability will be constructed to gauge the success of the business based on questions covering sales, expenses, and payroll amounts. To better understand the trajectories out of self-employment, individuals who had been operating a business after the date of random assignment but were no longer doing so at the time of the follow-up are asked about their reasons for doing so and what their major activity was afterward. Finally, to complement the findings from the implementation study, all individuals who had ever tried to become self-employed are asked about the challenges they faced in their endeavors.

Table A.3. Data Elements in the Follow-Up Survey, by Content Area and Availability of Similar Measures in Other SET Data Sources

Data Element of Follow-Up Survey by Content Area	Similar Baseline Measure Available in Application Package	Similar Measure Available in Program Participation Recordsa
Screening
Date of birth	
Last four digits of Social Security number	
Current Employment Status
Self-employment status	
Wage and salary (W/S) employment status	
Primarily work activity (if both self-employed and employed in W/S job)	
Reason for not working (if not employed)
Self-Employment Assistance Services
For specific types of services:
Whether received any at all		
Quantity received		
For all types of services:
Reason for not participating in any services (if applicable)
Business development topics addressed
Overall satisfaction
Organizations providing services
Business Development Milestones
Started or updated a business plan		
Applied for/received a business loan
Applied for/received a start-up grant		b
Received assistance in achieving milestone
Self-Employment Experiences
Ever tried to start a business
Ever self-employed	c
Number of business ventures startedd	c
Self-employment earnings over previous 12 months	
For current/most business venture:
Industry	
Whether business is incorporated	e
Number of employees	
Challenges faced in attempting to become self-employed
W/S Employment Experience
Ever had a job	c
Earnings over previous 12 months	
Typical hours worked	
Job Satisfaction, Fringe Benefits, Health Insurance, and Program Participation
Level of satisfaction with current employment situation
Unemployment insurance UI benefits:
Receipt with last 12 months
Received work search waivers
Program participation:
Received Trade Adjustment Assistance (TAA) benefits	
Received job placement or career counseling services from AJC or state labor exchange	
Received on-the-job training	
Received adult basic education	
Received supportive services including assistance with child care or transportation	
Contact Information	

Note: Items from the follow-up survey other than screening and contact information will be used to measure outcomes for the impact analysis. Baseline measures from the application package will be used in the impact analysis to form subgroups, to construct covariates to be included in multivariate regressions, and to track changes in outcomes over time. Selected baseline measures will also be used to screen applications, to describe the characteristics of the SET study population, and to conduct survey nonresponse analyses. Program participation measures will be used in the implementation study to provide quantitative information about the experiences of program participants.

^aProgram participation records will only be available for members of the treatment group and might only cover a subset of their self-employment service experiences.

^bProgram participation records will focus on applications for and receipt of seed capital microgrants available through the SET program.

^cFor these items, the baseline information form uses a five-year window prior to application.

^dBusiness ventures include owning a business and undertaking other self-employment business activity.

^eThe items are captured on the business idea form for the proposed business that the applicant wishes to start or expand through the SET demonstration.

Section D: Wage and salary employment. The survey gathers information about whether the respondent was employed in any wage or salary job since random assignment. Data collected about wage or salary earnings over the 12 months before the survey will be used together with comparable information about self-employment earnings to construct measures of total earnings. This section of the survey also asks for the number of weeks and number of hours worked over the prior year. This information will enable the evaluation to consider the extent to which receipt of demonstration services altered participants’ patterns of employment in the traditional wage and salary sector.

Section E: Job satisfaction and UI receipt. Respondents are asked to assess their satisfaction with their current employment situation at the time of the survey, which will provide qualitative information about the overall labor market impact of the SET Demonstration. In addition, this section collects information about the duration of UI benefit receipt. Finally, the survey assesses whether respondents received services though other government-sponsored workforce programs, such as the Trade Adjustment Assistance (TAA) benefits, job placement and career counseling services, job training, and supportive services. Gathering information about these outcomes for both the program and control groups will allow for a clearer understanding of the counterfactual condition and the extent to which SET services might crowd out or complement other services.

Section F: Updated contact information. Respondents are asked to check and, if necessary, revise the contact information that was collected at baseline so that the data collection team can seek clarification about the respondents’ answers, as needed, and ensure proper delivery of incentive payments (discussed in Section A.9). Collecting updated contact information is also important to facilitate further follow-up of the SET Demonstration participants should ETA sponsor such an effort. Although the current evaluation plans specify a follow-up of study participants to be conducted 18 months after random assignment, previous studies of self-employment programs clearly illustrate the importance of longer-term tracking of outcomes for these types of interventions (Benus et al. 1995, 2009). The evaluation’s follow-up survey has been designed to facilitate the collection of comparable information and estimation of impacts on key outcomes at later points in time if longer-term follow-up is conducted.⁸

3. Uses of Technology for Data Collection and to Reduce Burden

Advanced technology is being used in the evaluation’s follow-up survey to reduce burden on study participants.

Electronic advance letters describing the follow-up survey are being sent out to individuals who provide an email address on the application form (in addition to hard copies delivered in the postal mail). The electronic advance letters include a hyperlink to the survey website, which should reduce the effort and potential for error that could otherwise occur from typing the site address manually.

The follow-up survey is being conducted on the web to facilitate quick completion and submission. Data from web surveys are stored on secure servers and are immediately available and more accurate than self-administered questionnaires administered via paper and pencil. Web surveys reduce the amount of interviewer labor necessary to complete data collection and enable respondents to complete the questionnaire on their own schedule, in multiple sittings if needed, and without having to return any forms by mail. If an individual exits the application before completing it, they are given instructions on how to return to the web application and resume from the point they left, without any loss of data. A unique login and password are provided for them to protect their incomplete application data. Individuals who do not fill out a web survey and express an interest in completing it over the telephone will be administered the survey using computer-assisted telephone interviewing (CATI). To comply with Section 508 of the Rehabilitation Act, sample members likely to have difficulty completing a web survey will be offered the option of completing the survey by telephone by default.

Both self-administration via the web and the interviewer-administered CATI system reduce the respondent burden and costs compared with conducting in-person or paper-and-pencil interviews. Because the web survey is self-administered, it enables respondents to complete the survey on a schedule that is most convenient for them. Self-administration also offers the most cost efficiency because interviewers are not required. The web survey programming includes skip pattern logic, response code validity checks, specification of acceptable ranges, and consistency checks. As much information as possible is preloaded into the web survey in order to reduce respondent burden. An example of this is lists of local service providers from which respondents are able to choose. The web interface is easy to navigate to encourage sample members who open the web survey to continue through completion.

Any information preloaded in the web survey will also be preloaded into the CATI instrument to improve data accuracy and reduce respondent burden. CATI programs are efficient and accept only valid responses based on preprogrammed checks for logical consistency across answers. Interviewers are thus able to correct errors during the interview, eliminating the needs for costly call-backs to respondents. Further, CATI’s flexibility allows for the scheduling of interview times that are convenient for the sample member.

Both versions of the survey are expected to take approximately 20 minutes to complete. Except for language necessary to accommodate self-administration versus being asked by an interviewer, the content of both survey versions is identical.

4. Efforts to Identify Duplication

The SET follow-up survey provides unique information about the characteristics and outcomes of study participants that is crucial for conducting the impact analysis. No other survey data collection effort has been conducted or has been planned to collect similar information.

Administrative data systems provide very little individual-level data that can reliably estimate the effects of the program model on participant outcomes. There are two potential exceptions. First, earnings by employment status could be derived in some form based on data from the Internal Revenue Service (IRS) and/or Social Security Administration (SSA).⁹ Second, data on UI receipt could be obtained from state UI benefits records. However, collecting data on these measures using a survey is preferred for the following reasons:

• Earnings data from IRS and SSA are generally available over periods corresponding to one calendar year that will not, in general, coincide with the 12-month period after random assignment. This imperfect temporal overlap would reduce the precision of the study’s primary impact estimates. Further, research using matched administrative and survey data suggests that earnings information collected from a survey can, in some circumstances, yield an outcome measure that allows impacts to be more accurately estimated, compared with earnings information collected from an administrative source (Kapteyn and Ypma 2007). Thus, the follow-up survey is expected to provide data that will enable the evaluation to obtain more precise, and potentially more accurate, estimates of the SET Demonstration’s impacts on earnings.

• It is likely to be more cost effective to obtain information on UI benefit receipt via the application form and follow-up survey than by obtaining administrative UI benefit records from states. Substantial costs and burden would be incurred by seeking administrative UI benefits records corresponding to the 3,000 sample members included in the evaluation because the study team would have to negotiate with, and compensate, state UI agencies for the data. The data would also have to be thoroughly cleaned and checked to ensure that the records are properly matched to the sample members. Moreover, states may refuse to provide the data altogether. By contrast, including a few questions about UI benefit receipt will only very slightly increase the burden of the follow-up survey, which is already being fielded, to obtain other unique information about the sample members.

The follow-up survey provides information on other individual-level outcomes that are not measured in any existing data sources, including self-employment activities, economic self-sufficiency, and financial hardships. In addition, the follow-up survey will be the only source of data on utilization of self-employment services for both the program group and the control group.

5. Methods to Minimize Burden on Small Businesses or Entities

Some sample members will become self-employed and establish small businesses. Because investigating whether and how the demonstration services influence the outcomes of individuals interested in starting a business is the primary goal of the evaluation, these individuals are asked questions about their businesses and experiences receiving self-employment services. However, only the sample member and not other people in the business are asked questions, and the extent of the questions is limited to measures necessary to answer the main research questions of the SET Evaluation.

6. Consequences of Not Collecting the Data

The SET Evaluation represents an important opportunity for ETA to learn about a novel program for delivering self-employment services to dislocated workers who seek to pursue self-employment as a re-employment strategy. If the information collection requested by this clearance package is not conducted, policymakers and providers of self-employment services will lack high-quality information on the impacts of the SET Demonstration and whether it represents a program model that can feasibly be scaled up or appropriately modified to serve customers of the Federal workforce system.

The follow-up survey obtains information on a variety of important outcomes that could be affected by participation in the SET Demonstration. Many of these outcomes (for example, participation in self-employment services, specific self-employment activities, fringe benefits, health insurance, economic self-sufficiency, and financial hardships) cannot be measured in any existing data sources. The study’s primary outcomes (employment, earnings, and self-employment) are likely to be measured with lower accuracy in administrative data sources, as described in Section A.4. Thus, without the survey, the capacity of ETA to determine the impact of the SET Demonstration on participants’ outcomes would be severely limited.

Without the proposed extension, we will not be able complete surveys with all planned respondents for the follow-up survey. The additional time is needed to allow the 18-month follow-up window to pass for all sample members in the study. Without the extension, we estimate that we will only be able to attempt surveys with 25 percent of the demonstration applicants by the current expiration date of January 31, 2016. Assuming an 80 percent response rate, this would result in approximately 600 respondents (= 3,000 respondents × 0.80 response rate × 0.25 of study participants). This would give an overall response rate of 20 percent for the demonstration, greatly increasing non-response bias while reducing accuracy of impact estimates. Extending the deadline to September 30, 2017 will allow sufficient time to field the survey to all planned study applicants.

7. Special Data Collection Circumstances

There are no special circumstances surrounding data collection. All data will be collected in a manner consistent with Federal guidelines. There are no plans to require respondents to report information more than quarterly, to prepare a written response to a collection of information within 30 days of receiving it, to submit more than one original and two copies of any document, to retain records, or to submit proprietary trade secrets. The follow-up survey will produce valid and reliable results that can be generalized to the universe of the study and will include only statistical data classifications that have been reviewed and approved by OMB. It will include a pledge to maintain respondent privacy to the extent that existing statutes and regulations permit; the underlying disclosure and data security policies used by the contractor and DOL (see Section A.10) are consistent with the pledge. It will not unnecessarily impede sharing of data with other agencies for compatible use.

8. Federal Register Notice and Consultations Outside of the Agency

a. Federal Register Notice and Comments

As required by 5 CFR 1320.8 (d), a Federal Register notice, published on October 16, 2015 (80 FR 62572), announced plans to submit this data collection extension package to OMB. The Federal Register notice provided the public an opportunity to review and comment on the planned data collection extension within 60 days of the publication, in accordance with the Paperwork Reduction Act of 1995. No comments were received from the public. A copy of this 60-day notice is included as Appendix B.

b. Consultations Outside of the Agency

The data collection instruments have been developed based on the expertise of DOL and the contractor. DOL and the study team did not engage in any outside consultations for the follow-up survey.

9. Respondent Payments

Under the current OMB approval, the evaluation team conducted an incentive experiment to determine whether to offer sample members incentives for completing the follow-up survey, which are received in the form of a check after they complete the survey. The results from the experiment were presented to DOL and OMB. DOL determined that the best incentive scheme to use was to offer sample members $50 for completing within the first four weeks or $25 for completing thereafter. We will continue to use this respondent payment scheme during the proposed extension period.

The results of the incentive experiment show that the offer of incentives is important to gain cooperation from sample members; increase response rates; ensure the representativeness of the sample; and provide data that are complete, valid, reliable, and unbiased. ETA seeks to inform policy based on a data collection effort that meets high standards on these criteria, and offering incentives helps achieve that goal.

Substantial evidence on the benefits of offering incentives has become available. Incentives can help achieve high response rates by increasing the sample members’ propensity to respond (Singer et al. 2000). Studies offering incentives show decreased refusal rates and increased contact and cooperation rates. Among sample members who initially refused to participate, incentives increased refusal-conversion rates. By increasing sample members’ propensity to respond, incentive payments have been found to significantly reduce the number of calls required to resolve a case and the number of interim refusals. Thus, incentive payments can help contain costs and pass some of the costs of conducting the survey as a gain to the participant rather than into additional survey operations. The evidence gained during the SET incentive experiment supports this notion.

Our estimated cost of providing incentives for completion of the follow-up survey is $88,650 during the proposed extension period. This estimate assumes that 97 percent of survey completers will complete within the first four weeks and receive $50. The other three percent of completers will receive $25. We estimate that 1,800 completes (of 2,400 total) will occur during the proposed extension period of February 1, 2016 to September, 30 2017.

10. Privacy

This section contains a discussion of the measures that the evaluation team will take to safeguard the data that are part of this clearance request. The first subsection describes the contractor’s general policies for protecting privacy. The second subsection describes the contractor’s electronic security systems. The third section provides additional detail on the treatment of data with personally identifying information (PII) collected for this evaluation.

a. General Policies to Protect Privacy

This subsection describes, sequentially, the statements that will be made to study participants about privacy protection and the contractor’s staff training and clearance policies related to data security, and plans for a restricted-use data file to ETA (if produced as part of this study).

Statements about protecting respondent privacy. Sample members included in the follow-up survey will be ensured of the privacy of their responses as study researchers will implement administrative and security systems to prevent the unauthorized release of personal records. (These systems are discussed in detail in the following subsections.) The agency also gave the public notice of the planned evaluation through publication in the Federal Register (see Appendix B). All respondent materials will include assurances of privacy protection. These include letters sent to sample members and information posted on the web site for the SET follow-up survey. In addition, as part of the telephone interviewers’ introductory comments, sample members will be told that their responses are private and will have the opportunity to have their questions answered. Interviewers are trained in procedures to maintain privacy and are prepared to describe them in full detail, if needed, or to answer any related questions raised by participants. For example, the interviewer will explain that the individual’s answers will be combined with those of others and presented in summary form only.

Staff training and clearance policies. All data items that identify sample members will be kept only by the evaluation contractor, Mathematica, for use in assembling data and in conducting the follow-up survey. (As discussed in greater detail below, any data delivered to ETA will not contain personal identifiers, thus precluding individual identification.) It is the policy of Mathematica to efficiently protect private information and data in whatever medium it exists, in accordance with applicable Federal and state laws and contractual requirements. In conjunction with this policy, all Mathematica staff will do the following:

1. Comply with a Mathematica pledge that is signed by all full-time, part-time, and hourly Mathematica staff, and with the Mathematica Security Manual procedures to prevent the improper disclosure, use, or alteration of private information. Staff may be subjected to disciplinary, civil, or criminal actions for knowingly and willfully allowing the improper disclosure or unauthorized use of private information.

2. Access private and proprietary information only in performance of assigned duties.

3. Notify their supervisor, the project director, and the Mathematica Incident Response Team if private information has been disclosed to an unauthorized individual, used in an improper manner, or altered in an improper manner. All attempts to contact Mathematica staff about any study or evaluation by individuals who are not authorized access to the private information will be reported immediately to both the cognizant Mathematica project director and the Mathematica security officer.

In addition, the evaluation team members working with the data for this study will have previously undergone background checks. These may include filling out an SF-85 or SF-85P form, for example, authorizing credit checks, and having fingerprints taken.

Restricted Use Data Files. To facilitate external verification and replication of the study findings, as well as additional research, the evaluation team will consider producing restricted use data files containing key analysis variables created for the SET evaluation at the end of the study. (Current study plans do not provide for creation of such restricted use files.) If produced, these data delivered to ETA will not contain personal identifiers, thus precluding individual identification. These data files will follow the relevant OMB checklist to ensure that they can be distributed to authorized researchers with appropriate restrictions. Steps would also be taken to ensure that sample members cannot be identified in indirect ways. For example, categories of a variable will be combined to remove the possibility of identification due to a respondent being one of a small group of people with a specific attribute. Variables that would be carefully scrutinized include age, race and ethnicity, household composition and location, dates pertaining to employment, household income, household assets, and others as appropriate. Variables would also be combined in order to provide summary measures to mask what otherwise would be identifiable information. Although it cannot be predicted which variables will have too few respondents in a category, the SET evaluation contractor would not report categories or responses that are based on cell sizes of fewer than five. If necessary, statistical methods would be used to add random variation within variables that would be otherwise impossible to mask. Finally, variables that could be linked to identifiers by secondary users would be removed or masked.

b. Systems Security

Mathematica’s computer facilities include state-of-the-art hardware and software. The hardware and software configurations have been designed to facilitate the secure processing and management of both small- and large-scale data sets.

Facility. The doors to Mathematica’s office space and Survey Operations Center (SOC) are always locked and require a key card to gain entry. All SOC staff are required to display current photo identification while on the premises. Visitors are required to sign in and out and must wear temporary ID badges while on the premises. Any network server containing private data is located in a controlled, limited-access area. All authorized external access is through a server under strict password control. The SOC features lockable storage areas for sensitive documents, and controlled access to computerized files and systems.

Network. Sensitive data are stored in secure folders that reside on a Windows 2008 Server volume using Microsoft NT File System (NTFS). BitLocker encryption software, configured to use a 256-bit advanced encryption standard (AES) key, encrypts data on the volume as they are stored. The encryption persists for the life of the volume. NTFS/BitLocker makes the data accessible only to users with authorized access, and makes data inaccessible to software that circumvents normal access control, in case the media are stolen. NTFS/BitLocker stores user data in an encrypted format on the volume, but it works transparently with most applications and backup utilities. All the rules of file system trustee assignments, trustee rights, ownership, sharing, visibility, locking, transactions, and space restrictions remain the same on the encrypted volume. Data in the “Secure_Data” folders are backed up using ArcServe 11.5, which encrypts the contents using the 3DES algorithm. These separate backups are overwritten every two months by backups of newer secure data, a process that enables compliance with secure data destruction requirements.

Access to all network features, such as software, files, printers, Internet, email, and peripherals, is controlled by user ID and password. Mathematica staff are required to change their passwords for computer access no fewer than every thirty days, and passwords must be at least eight characters long and contain characters from three of the following four categories: uppercase letters, lowercase letters, numerals, and non-alphabetic characters. All user IDs, passwords, and network access privileges are revoked within one working day for departing staff and immediately for terminated staff. All staff are required to log off the network before leaving for the day.

Printers. Printer access is granted to all staff with a valid user ID and password. The physical hard disks on which the printer queues reside are subject to the same security and crash procedures that apply to the file servers. Printer queues are confined to write-access to all staff. No staff have read-access to the printer queues; that is, they cannot browse the contents of the printer queues. Printer stations are appropriately monitored according to the sensitivity of the printed output produced. No private or proprietary data or information can be directed to a printer outside Mathematica’s offices. Staff are instructed not to print sensitive data if possible and to retrieve printouts containing such data immediately.

Electronic communication. Each of Mathematica’s locations has a site-specific local-area network (LAN). A combination of T1 and ethernet private line (EPL) lines links the site-specific LANs into a wide-area network (WAN) and supports cross-office communications. Traffic on the Mathematica internal network, which is not encrypted, is secured by these links, all of which are private, point-to-point communication lines dedicated to Mathematica traffic and completely contained within Mathematica’s firewalls. Because each office is connected to other offices solely by these private point-to-point lines and not through the Internet, all WAN traffic is contained and protected within Mathematica’s firewalls; no WAN traffic is routed through the Internet.

c. Treatment of Data with Personal Identifying Information Collected for the Evaluation

All data containing PII—including SSN, name, home address, date of birth, and telephone number—are considered to be sensitive, or private data. The SET Evaluation is in compliance with the aforementioned company security policies. In this subsection, study procedures for storing and processing PII are described, followed by a discussion of additional considerations for the PII associated with specific data collection and management activities.

1) Procedures for Handling PII

Data files. When possible, electronic files for everyday use are created without personal identifiers. Data and sample files that must contain sensitive data are stored and analyzed on one of Mathematica’s secure hard-drives. Specifically, staff working on this project will be instructed to maintain all files with private data in project-specific, encrypted folders on the Mathematica network. Access control lists restrict access on a need-to-know basis and only to project staff who are specifically authorized to view the sample data (as designated by the project director or survey director) to select and process the sample or to process the data files. Sensitive data that are no longer needed in the performance of the project will be magnetically erased or overwritten using Hard Disk Scrubber or equivalent software, or otherwise destroyed.

Access. Electronic files with private data will be stored in restricted-access network directories. Access to restricted directories is limited through access control permissions, on a need-to-know basis to staff who have been assigned to and are currently working on the project. When temporarily away from their work areas, project staff are instructed to close files and applications and to lock their workstations using the CTRL-ALT-DEL command. Workstations automatically lock within a set number of minutes and a password must be used to regain access through the protected screen saver.

Electronic communication. For internal emails, staff are forbidden to transmit sensitive study information as a regular file attachment; they are instructed instead to use the “insert hyperlink” feature in Outlook to include a shortcut to the file. This enables the receiver to go to the file directly if authorized, but will not allow access to unauthorized individuals. In addition, staff are instructed to avoid including sample member names or any other PII in internal emails, so that there is no potential for these to be viewed by others.

Emails sent outside Mathematica are not automatically encrypted, and therefore neither the text nor attachments are secure. Before sending an email containing sensitive information, the sender is obligated to ensure that the recipient is approved to receive such data. When files must be sent as attachments outside Mathematica, staff are instructed to use WinZip 14.5 (256-bit AES encryption) to password-protect the file and transmit the password to the recipient using a separate form of communication, preferably via telephone. When a sample member’s name and contact information are sent outside Mathematica, the information is included in a secure attachment rather than in the text of the email.

Hard-copy printouts. Sensitive temporary work files, used to create hard-copy printouts and stored in temporary work files on local hard drives, are deleted on a periodic basis. Hard-copy output with private information is shredded or stored securely when no longer needed. Test printouts of data records carrying personal identifiers that are generated during file construction are shredded.

Incident response. Staff are instructed to report any incidents or potential incidents involving PII to Mathematica’s Incident Response Team immediately by email or using an internal reporting web site. When notified, the Incident Response Team determines whether an incident has occurred and has to be reported to ETA. If so, the Incident Response Team informs ETA of the incident within one hour of its discovery.

2) Additional Considerations for PII Associated with Specific Data Collection Activities

Follow-up survey and case identification numbers. Follow-up surveys are submitted electronically. Applicants include their SSN as a digital “signature” indicating consent. Sample members who initiate a follow-up survey are asked to confirm the last four digits of their SSN. To protect this sensitive PII, surveys will be completed and submitted using a secure web-based interface. Mathematica will process and store the results using secure servers consistent with the systems security policies described above. Each sample member will be assigned a unique case identification number that may be used to link all of the study data files together.

Telephone interviewing and locating for follow-up survey. Telephone interviewers for the SET follow-up survey are seated in a common supervised area. As part of the process to verify that the correct sample members have been reached, interviewers have access to respondents’ names and birthdates, as well as the last four digits of their SSNs. Birth date and the last four SSN digits will be displayed on the computer screen only temporarily, at the beginning of the survey, so that the interviewer can verify the sample member’s identity. Interviewing staff for this project receive training that includes general SOC security and privacy procedures, as well as project-specific training that includes explanation of the highly private nature of this information, instructions to not share it or any PII with anyone not on the project team, and warnings about the consequences of any violations. After receiving training, these staff sign privacy and nondisclosure agreements. Telephone interviews are recorded for educational and training purposes only, to aid SOC staff in improving their interviewing skills.

Staff who work on updating sample members’ contact information when the original contact is not successful must have access to key identifying information for short periods. These staff members receive training that includes general SOC security and privacy procedures, as well as project-specific privacy training that includes clear instructions on what data and databases can be accessed and what data are required and can be recorded. After receiving training, these staff sign privacy and nondisclosure agreements.

Locators may talk to sample member’s family, relatives, or other references to obtain updated contact information. To protect the sample member, locators are given scripts on what they can and cannot say when using these sources to obtain information. For example, they are instructed not to tell anyone that the sample member has been selected to participate in a study of people receiving self-employment training assistance. Rather, they indicate that Mathematica is trying to reach the sample member for an important study sponsored by ETA. Postcards describe the need to speak to the person who agreed to participate in the study.

In addition, locating staff keep only the minimum amount of printed personal information needed to perform assigned duties. Hard-copy materials (such as locating or calling contact sheets) containing data with any individual identifiers (for example, name and street address) are stored in a locked cabinet or desk when not being used. When in use, such materials are carefully monitored by a project supervisor and are never left unattended. At the conclusion of the project, a final disposition of all remaining sample will be made, and contact sheets and other associated materials will be destroyed.

11. Questions of a Sensitive Nature

The follow-up survey contains some questions that could be considered sensitive. These questions are related to earnings, income, financial hardships, and the receipt of public assistance. Depending on an individual’s circumstances, any of these questions could be perceived as sensitive. All respondents are informed that they can decline to answer any question they do not wish to answer.

All questions in the SET follow-up survey, including those deemed potentially sensitive, have been pretested, and many have been used extensively in prior surveys with no evidence of harm. Questions about income, financial hardships, and receipt of public assistance are necessary to measure the economic well-being of study participants. Obtaining information about these potentially delicate topics is integral to addressing the research questions posed by the study, in order to describe the characteristics of SET participants, describe their outcomes, and assess the impact of the SET program.

12. Estimated Hour Burden of the Collection of Information

The proposed extension will not increase the total burden of the follow-up survey. The total burden for the follow-up survey was estimated in the non-substantive change request (ICR reference number 201408-1205-005) at 800 hours at an indirect total cost burden of $13,824 spread over an 18-month data collection window. These estimates were annualized to 533 burden hours and a $9,210 cost burden. The non-substantive change request assumed 2,400 survey completes (80 percent response rate × 3,000 SET applicants) over this 18-month period.

This new request estimates a total of 2,400 completed surveys completed over a period from April 2015 to September 2017. We expect to complete 600 surveys prior to January 31, 2016 and 1,800 during the proposed extension period (February 2016 to September 2017). The survey is estimated to take 20 minutes to complete, with a total burden of 800 hours for all follow-up data collection. We estimate a total of 200 burden hours will fall before February 1, 2016 and the remaining 600 hours after during the extension period.

Table A.4 presents the number of respondents, the number of responses per respondent, the average burden hours per response, and the total annual burden hours for the follow-up survey data collection that will occur during the extension period. We expect to complete 20 minute web surveys with 1,800 people for a total of 600 burden hours, annualized to 360 hours. Table A.5 presents annualized estimates of indirect costs to all respondents for the follow-up survey data collection instrument during the proposed extension. At an average wage of $17.28¹⁰ per hour the cost estimate for this customer burden is $6,221 (360 annual burden hours at $17.28 per hour).

Table A.4. Annual Burden Estimates for SET Demonstration Follow-up Survey, February 1, 2016 to September 30, 2017

Activity	Number of Respondents	Frequency	Total Annual Responses	Time Per Response	Total Annual Burden (Hours)	Hourly Rate*	Monetized Value of Respondent Time
Follow-up Survey	1,800	Once	1,080	20 min.	360	$17.28	$6,221
Unduplicated Totals	1,800	Once	1,080	20 min.	360	$17.28	$6,221

*Source: Hourly wage rates were calculated using the public use dataset for the Growing America Through Entrepreneurship (GATE) demonstration based on members of the study’s control group whose characteristics at baseline were similar to the criteria used to identify dislocated workers for the SET Demonstration https://www.doleta.gov/reports/projectgate/. At the 18-month follow-up survey (the midpoint of which was March 2006), the average wage rate among employed members of this GATE subgroup was $14.62, which translates to $17.28 in 2014 dollars after adjusting for inflation. $17.28 x 360 hours = $6220.80.

13. Estimated Total Annual Cost Burden to Respondents and Record Keepers

There will be no start-up or ongoing financial costs incurred by respondents that result from the data collection efforts of the SET Evaluation. The proposed information collection plan will not require the respondents to purchase equipment or services or to establish new data retrieval mechanisms.

14. Estimated Annualized Cost to the Federal Government

The contractor will incur a cost of $3,200,000 when carrying out the study over a 68-month (5.7-year) period, for an annualized cost of $564,373.89. The total cost of the study will not change with the proposed extension; thus, the annualized cost will decrease. Of these expenses:

• $342,495 is for development and conduct of the evaluation’s random assignment procedures, corresponding to an annualized cost of $60,404.76;

• $56,508 is for development, testing, and maintenance of the management information system for program participation records, corresponding to an annualized cost of $9,966.14;

• $396,664 is for the administration of the follow-up survey, corresponding to an annualized cost of $69,958.38; and

• $250,727 is for conduct of the implementation study site visits and case study interviews, corresponding to an annualized cost of $44,219.93.

15. Changes in Burden

The proposed extension will not change the total amount of burden associated with the administration of the follow-up survey. Total burden is expected to remain at the approved amount of 800 (2,400 interviews × 20 minutes per interview).

16. Publication Plans and Project Schedule

The data collection for which this Supporting Statement is seeking clearance will not result in publicly available records. However, data collected from the baseline applications and follow-up surveys may be made available by ETA to authorized researchers through restricted use data files, if such data files are produced at the conclusion of the study. Data and study progress will be documented internally throughout the project.

The evaluation plan includes a range of deliverables and reports. Table A.6 shows an outline of these deliverables.

Table A.6. Deliverable Time Line

Deliverable	Date
Demonstration Procedures Manual	January 2013
Design Report	December 2015
Issue Briefs (2)	December 2015
Final Report	June 2018

17. Reasons for Not Displaying Expiration Date of OMB Approval

The expiration date for OMB approval will be displayed on all forms distributed as part of the data collection.

18. Exceptions to the Certification Statement

Exception to the certification statement is not requested for the data collection.

REFERENCES

Baptista, Rui, Murat Karaöz, and Joana Mendonça. “Entrepreneurial Backgrounds, Human Capital, and Start-Up Success.” Jena Economic Research Papers. Jena, Germany: Friedrich-Schiller University and the Max Planck Institute of Economic, 2007.

Bellotti, Jeanne M., Sheena M. McConnell, and Jacob Benus. “Growing America Through Entrepreneurship: First Findings from Project GATE.” Princeton, NJ: Mathematica Policy Research, August 2006.

Benus, Jacob M., Terry R. Johnson, Michele Wood, Neelima Grover, and Theodore Shen. “Self-Employment Programs: A New Reemployment Strategy: Final Report on the UI Self-Employment Demonstration.” Unemployment Insurance Occasional Paper 95-4. Washington, DC: U.S. Department of Labor, Employment and Training Administration, 1995.

Benus, Jacob, Sheena M. McConnell, Jeanne M. Bellotti, Theodore Shen, Kenneth N. Fortson, and Daver Kahvecioglu. “Growing America Through Entrepreneurship: Findings from the Evaluation of Project GATE.” Final report submitted to the U.S. Department of Labor, Employment and Training Administration. Columbia, MD: IMPAQ International, LLC, May 2008.

Benus, Jacob, Theodore Shen, Sisi Zhang, Marc Chan, and Benjamin Hansen. “Growing America Through Entrepreneurship: Final Evaluation of Project GATE.” Final report submitted to the U.S. Department of Labor, Employment and Training Administration. Columbia, MD: IMPAQ International, LLC, December 2009.

Calliendo, Marco, Frank Fossen, and Alexander Kritikos. “The Impact of Risk Attitudes on Entrepreneurial Survival.” Journal of Economic Behavior and Organization, vol. 76, no. 1, October 2010, pp. 45–63.

Evans, David S., and Linda S. Leighton. “Some Empirical Aspects of Entrepreneurship.” American Economic Review, vol. 79, no. 3, June 1989, pp. 519–535.

Fairlie, Robert W., and Alicia Robb. Race and Entrepreneurial Success: Black-, Asian-, and White-Owned Businesses in the United States. Cambridge, MA: MIT Press, 2008.

Harada, Nobuyuki. “Who Succeeds as an Entrepreneur? An Analysis of the Post-Entry Performance of New Firms in Japan.” Japan and the World Economy, vol. 15, no. 2, April 2003, pp. 211–222.

Kapteyn, Arie, and Jelmer Y. Ypma. “Measurement Error and Misclassification: A Comparison of Survey and Administrative Data.” Journal of Labor Economics, vol. 25, no. 3, July 2007, pp. 513–551.

Kosanovich, William T., and Heather Fleck. “Comprehensive Assessment of Self-Employment Assistance Programs.” ETA Occasional Paper 2002-01. Washington, DC: U.S. Department of Labor, Employment and Training Administration, 2002.

Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. “Experiments with Incentives in Telephone Surveys.” Public Opinion Quarterly, vol. 64, no. 2, summer 2000, pp. 171–188.

This page has been left blank for double-sided copying.

1 To receive training services under Title I of WIA, a dislocated worker is an individual who (1) (A) has been terminated or laid off or has received a notice of termination or layoff from employment , and (B) (a) is eligible for or has exhausted unemployment insurance or (b) has demonstrated an appropriate attachment to the workforce, but is not eligible for unemployment insurance, and (C) is unlikely to return to a previous industry or occupation; (2) has been terminated or laid off or received notification of termination or layoff from employment as a result of a permanent closure or substantial layoff, or is employed at a facility where the employer has made the general announcement that the facility will close within 180 days; (3) was self-employed but is unemployed as a result of general economic conditions in the community or because of a natural disaster; or (4) is a displaced homemaker who is no longer supported by another family member. Individuals are considered eligible for the SET Demonstration if they meet any of these four qualifications, irrespective of whether they register for staff-assisted services with a WIA American Job Center.

2 Title I of WIA, Section 171(b) states that DOL shall “… through grants or contracts, carry out demonstration and pilot projects for the purpose of developing and implementing techniques and approaches, and demonstrating the effectiveness of specialized methods, in addressing employment and training needs.” Section 172 grants DOL the authority to evaluate the activities authorized under Section 171, and Section 172(c) specifies that the agency “shall utilize appropriate methodology and research designs, including the use of control groups chosen by scientific random assignment methodologies.”

3 SEA participants also receive an allowance of equal value in lieu of their UI benefit and are not subject to work-search requirements, as long as they are engaged full-time in qualifying activities related to starting their business. These SEA allowances are not diminished by any self-employment earnings. Similar allowances were provided in selected sites participating in the SET Demonstration.

4 In June 2008, ETA awarded GATE II grants to the state workforce agencies in Alabama, Minnesota, North Carolina, and Virginia to provide self-employment assistance to older workers and workers in rural areas. However, a discussion of the GATE II project is not included here because no findings have been released as of this writing.

5 Seed capital offered through the Washington demonstration project differed from the microgrants offered to SET participants in three important ways. First SEED offered these funds in a way that simulated a “cash out” of a participant’s UI entitlement. In the SET program, microgrants are independent of an individual’s UI benefits receipt. Second, the average lump-sum payment received by SEED participants was $7,129 in 2012 dollars, which is substantially greater than the maximum allocation of $1,000 per SET participant. Third, in addition to meeting business milestones and program participation, the SEED program specified that lump-sum payments be given to participants who had already obtained “adequate financing” (Benus et al. 1995, page iv). The SET program, by contrast, does not require participants to have preexisting financing. Instead, a potential use of the seed capital microgrants is to better position SET participants to attract subsequent loans, capital grants, or investments.

6 As such, the study population consists of a purposively selected quota sample recruited from a broader population of interested individuals who self-select into SET orientation sessions. There is no burden imposed on the broader population of individuals participating in orientations, since no information will be collected from them. However, as noted below in this section, it is anticipated that the burden associated with completing study application materials could be incurred by up to 4,000 applicants in order to meet the study’s enrollment target of 3,000 individuals going through random assignment. The implications of the study selection process for interpreting the study’s statistical findings are described in Part B of this package.

7 U.S. Small Business Administration. “Credit Factors.” Washington, DC: SBA, n.d. Available at http://www.sba.gov/category/navigation-structure/loans-grants/small-business-loans/application-process/credit-factors. Accessed November 10, 2011.

8 For instance, Sections A, C, D, and E of the survey (or portions of these sections) could be slightly modified to reflect a longer follow-up period—for example, 3, 5, or 8 years after random assignment—to support the estimation of longer-term impacts of the SET program. A separate OMB clearance request would be submitted for any modified data collection instruments associated with a longer-term evaluation.

9 Wage records from state administrative UI systems do not cover self-employment earnings.

10 Hourly wage rates were calculated using the public use dataset for the Growing America Through Entrepreneurship (GATE) demonstration based on members of the study’s control group whose characteristics at baseline were similar to the criteria used to identify dislocated workers for the SET Demonstration. At the 18-month follow-up survey (the midpoint of which was March 2006), the average wage rate among employed members of this GATE subgroup was $14.62, which translates to $17.28 in 2014 dollars after adjusting for inflation.