STED-ETJD Supporting Statement Part A Final 6-26-15

STED/ETJD

OMB Supporting Statement: Part A



SUBSIDIZED AND TRANSITIONAL EMPLOYMENT DEMONSTRATION (STED) AND ENHANCED TRANSITIONAL JOBS DEMONSTRATION (ETJD)

SUPPORTING STATEMENT A

REQUEST FOR OMB CLEARANCE

OMB No.: 0970-0413

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

October 2012

Revised May 2013¹

Revised March 2014²

Revised June 2015³

TABLE OF CONTENTS

Part A. Justification 1

A1. Circumstances Necessitating Data Collection 1

A.1.1 Background and Policy Context 1

A.1.2. Overview of the STED and ETJD Evaluations 7

A.2. How, by Whom, and for What Purpose the Information is to be Used 8

A.3. Use of Information Technology for Data Collection to Reduce Respondent Burden 26

A.4. Efforts to Identify Duplication 27

A.5 Burden on Small Business 28

A.6 Consequence If Data Collection is Not Conducted 28

A.7 Special Data Collection Circumstances 28

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission 28

A.9 Justification for Respondent Payments 29

A.10 Assurances of Privacy 31

A. 11 Questions of a Sensitive Nature 31

A.12 Estimates of the Hour Burden of Data Collection to Respondents 31

A.13 Other Cost Burden or Respondents and Record Keepers 36

A.14 Annualized Cost to the Government. 36

A.15 Reasons for Any Program Changes or Adjustments 36

A.16 Tabulation, Analysis, and Publication Plans and Schedule 37

A.17 Reasons for Not Displaying OMB Approval Expiration Date 38

A.18 Exceptions to Certification Statement 38

Part B. Collection of Information Using Statistical Methods 39

B.1 Respondent Universe, Sampling Selection, and Expected Response Rates 39

B.2 Procedures for Data Collection and Statistical Analysis 40

B.3 Maximizing Response Rates and Issues of Nonresponse 44

B.4 Pre-Testing 50

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 50

LIST OF EXHIBITS

1.1 Conceptual Framework for the STED and ETJD Evaluations 6

1.2 Annual Burden Estimates for Previously Approved Instruments 32

1.3 Exhibit 1.3: Annual Burden Estimates for New Instrument 35

1.4 Exhibit 1.4: Total Annual Burden for OMB #0970-0413 35

2.1 Follow-Up Survey Sample Sizes 39

2.2 Minimum Detectable Effects, Per Site, Per Survey 42

2.3 Interim Response Rates 47

LIST OF APPENDICES

1. Background on Subsidized and Transitional Employment Programs and Evaluations (previously reviewed)

2. Informed Consent Form (previously reviewed and approved)

3. Contact Information Form (previously reviewed and approved)

4. Baseline Information Form (previously reviewed and approved)

5. Tracking Letters (previously reviewed and approved)

6. Program Staff Implementation Questionnaire (previously reviewed and approved)

7. Participant Implementation Questionnaire (previously reviewed and approved)

8. Worksite Supervisor Implementation Questionnaire (previously reviewed and approved)

9. Employer Implementation Questionnaire (previously reviewed and approved)

10. Interview Guides (previously reviewed and approved)

11. Participant Focus Group Discussion Guide (previously reviewed and approved)

12. Worksite Observation Guide (previously reviewed and approved)

13. Program Staff Time-Use Worksheet (previously reviewed and approved)

14. Cost Data Collection Protocol (previously reviewed and approved)

15. Six-month Follow-Up Survey (previously reviewed and approved)

16. 12-month Follow-Up Survey (previously reviewed and approved)

17. List of Technical Working Group (TWG) members (previously reviewed)

18. 60-Day Federal Register Notice

19. Implementation Questionnaire Screenshots (previously reviewed)

20. Overview of the 30-month Follow-Up Survey (previously reviewed)

21. 30-month Follow-Up Survey (previously reviewed and approved)

22. Overview of the June 2015 Revisions

23. Informed Consent Forms for Paycheck Plus site

W1. MDRC Informed Consent Form

W2. IRS Consent to Disclose Form

W3. IRS Consent to Use Form

24. Contact Information Form for Paycheck Plus site

25. Baseline Information Form for Paycheck Plus site

26. Staff Script – Talking with Clients About the Paycheck Plus Study

SUPPORTING STATEMENT

Subsidized and Transitional Employment Demonstration (STED)

And

Enhanced Transitional Jobs Demonstration (ETJD)

PART A. Justification

A1. Circumstances Necessitating Data Collection

This is a revision to the Subsidized and Transitional Employment Demonstration (STED) and Enhanced Transitional Jobs Demonstration (ETJD) information collection. This request includes a contact information form and a baseline form to be administered to study participants in a new STED site. As discussed below, these data collected for the new site will provide important demographic information about the study sample, help to ensure the comparability of the program and control groups, and allow for an analysis of program effects for selected subgroups.

This Supporting Statement provides information for a combined data collection effort to determine the effectiveness of subsidized and transitional employment programs being evaluated by the Administration for Children and Families (ACF) Subsidized and Transitional Employment Demonstration (STED) and the U.S. Department of Labor Enhanced Transitional Jobs Demonstration (ETJD). Both of these evaluations are being conducted by MDRC and its subcontractors MEF Associates, Branch Associates, D.I.R. Inc., and Abt Associates on behalf of ACF and ETJD. The evaluation consists of eight STED sites and seven ETJD sites; however, since two of the programs funded through the ETJD project are being evaluated through the STED project, a total of thirteen sites are participating in the two projects. All seven of the sites that are part of the ETJD project (including two which are being evaluated as a part of STED) have completed enrollment of participants into the evaluation. Five of the STED sites have also completed enrollment, two are currently enrolling participants, and anew site is projected to start enrollment in the near future. This information collection request is regarding a new STED site that will test the effects of an enhanced Earned Income Tax Credit for low-income individuals without dependent children.

The primary goal of both projects is to increase our knowledge about the efficacy of programs aimed at helping target populations secure and maintain unsubsidized employment. ACF and ETA have agreed to collaborate on the design of data collection instruments to promote consistency across the project.

A.1.1 Background and Policy Context

Subsidized employment programs aim to use public funds to create or support work opportunities for people who would not otherwise be employed. A variety of different subsidized employment models have been implemented or tested in the U.S. since the 1930s. Most subsidized employment programs are designed, at least in part, to provide work-based income support to people who cannot find regular jobs. During some economic downturns, subsidized employment programs have operated on a large scale and served a fairly broad range of people, such as the New Deal’s Works Progress Administration, the 1970s Public Service Employment (PSE) program, and some of the state programs operated using the Temporary Assistance for Needy Families Emergency Contingency Fund in 2009-2010. More commonly, subsidized employment programs have been relatively small and narrowly targeted to people who have difficulty finding work even when the labor market is strong. Some of these programs aim to achieve important indirect goals that are seen as linked to higher employment, such as reduced recidivism for former prisoners, while others have focused on improving public facilities or infrastructure, or providing community services.

A subset of subsidized employment programs have explicitly aimed to build the employability of economically disadvantaged populations. In these programs, the goal is not only to provide work and income support in the short-term, but also to improve participants’ performance in the regular labor market over the longer-term. Past evaluations of subsidized and transitional employment approaches that share this goal have produced mixed results. Yet, the prevalence, popularity, and potential opportunity of these approaches warrant further study to address questions that remain unanswered. Appendix A discusses the variations in subsidized employment approaches in more detail and highlights the major findings from a variety of studies examining these interventions. It is within this context of increased interest and unanswered research questions that the Administration for Children and Families and the Employment and Training Administration are conducting these current studies (i.e., STED and ETJD).

Both studies will examine the impact of variations in subsidized and transitional employment programs on the outcomes of the targeted populations. The basic research questions for both studies are:

• What effects do subsidized and transitional jobs programs targeted at particularly vulnerable clients have on their populations of interest, such as non-custodial parents, TANF recipients, low-income youth transitioning to the labor force, prisoners re-entering the community, low-income individuals with disabilities, American Indians and Alaska natives, and English Language Learners?

• Do different subsidized employment approaches, i.e., length of employment or content of program, lead to different impacts for participants? Further, what are the effects of programs that place clients directly into subsidized jobs in the regular labor force?

• How can existing subsidized and transitional employment models be adjusted to promote longer-term outcomes for participants?

• How might subsidized employment programs adjust their engagement techniques to promote continued participation and improved employment outcomes for populations facing different challenges to non-subsidized work?

• What effect might a “tiered” subsidized employment model that serves a wide range of people and allocates slots to those who need them most have on clients?

• What are the outcomes for participants in a model that includes occupational skills training or sectoral targeted training?

The new STED site also aims to improve the economic circumstances of a disadvantaged population – low income individuals without dependent children—while promoting work. By encouraging work and providing a work-based supplement, the new site fits broadly within the goals of the STED project.

One of the most documented changes in the labor market over the past several decades has been the dramatic increase in earnings inequality.⁴ Workers with a college education have seen their wages increase by 12 percent since the early 1970s, while wages for those without a high school diploma have fallen by 20 percent.⁵ The fall in wages is concentrated entirely among men. Men with less than a high school education earned on average $17.45 per hour in 1975 (in 2011 dollars), compared with only $12.71 in 2011, a 27 percent decline.⁶

One result of these trends is that many U.S. workers can no longer support a family, even working full time. Another consequence of falling wages, however, is that fewer men are participating in the labor market, or at least the formal labor market. The labor force participation rate for all men fell from 75 percent in 1990 to 71 percent in 2010.⁷ Employment rates are especially low for men with the lowest education levels.⁸

There have been several recent proposals to counter these trends by providing expanded EITC benefits to less-skilled, childless workers, driven in part by the success achieved from the EITC expansions for low-income parents.⁹ The EITC for parents has been expanded considerably since the early 1990s and has been found to have positive effects on single parents’ employment rates and income and on their children’s well-being.¹⁰ It has had minimal effect on individuals without dependent children, given that benefits are nominal.

The new STED site is a test of the idea to expand the credit for this group. As with the existing STED programs, the new site provides work-based income support, although in regular, not transitional, jobs. The new site has similar goals as the existing STED sites, to increase the employability and income, and it target many of the same groups – young adults, former prisoners, and non-custodial parents.

The key research questions for this site include:

• What is the take up rate among eligible participants?

• How does the expanded EITC affect employment and earnings?

• How does the expanded EITC affect secondary outcomes, such as benefit receipt and child support payments?

A Conceptual Framework for the ETJD and STED Projects

The ETJD and STED projects will build from, and aim to improve on, the logic model that underlies subsidized employment programs and on the evaluation results described in Appendix A. Because many of the results have been mixed or discouraging, it is important to look critically at the logic model to identify assumptions that should be reexamined. This process can lead to hypotheses about what might work better, and these hypotheses, in turn, may help to identify the types of models that should be tested.

Most of the subsidized employment programs that have been rigorously tested have assumed that, for certain types of hard-to-employ individuals, structured work experience can:

• Build a work history, increasing the odds that participants can get regular jobs. The models assume that employers will value participants’ experience in the subsidized jobs when evaluating job candidates (in On the Job Training-like models, the participants’ subsidized work experience occurs in the firm that is considering hiring him or her).

• Teach soft skills so that participants are better prepared to hold jobs. The models assume that people are best able to learn to work by working. (Few of the subsidized employment models tested to date have focused on hard skill development).

• Give program staff an opportunity to identify and address workplace problems. The models assume that staff can do this best by observing participants on the job rather than by relying on paper-and-pencil assessments.

• Give people an incentive to stay active in programs. The models assume that participants will be more likely to stay engaged when they are being paid.

The fact that most of these programs have been unable to boost unsubsidized employment suggests that at least some of these assumptions may be flawed. Possible explanations for the generally disappointing results – derived from a mixture of evaluation evidence and practitioner hunches – include the following:

• The programs have not been properly targeted. It is possible that there are groups who could benefit from subsidized employment models, but they have not been targeted in the programs that have been studied – or have been served in small numbers so that subgroup impacts could not be detected. There are hints from past transitional jobs evaluations that those programs may achieve stronger impacts for less employable participants.

• Employers do not value subsidized work experience in program settings. It is possible that, when employers evaluate job candidates, a few weeks or months of experience in a “program” job may not be sufficient to overcome skills deficits or negative perceptions of welfare recipients, former prisoners, or other disadvantaged groups. Perhaps employers would feel differently if the work experience took place in their own worksite.

• Learning soft skills is not enough. Perhaps programs would do better if they also taught occupational skills to help people qualify for higher paying jobs. Some argue that these skills are also taught better in a hands-on work environment. Alternatively, perhaps earnings supplements or other incentives are needed to encourage people to keep low paying unsubsidized jobs. A number of other studies have found positive results for earnings supplements¹¹ and MDRC is testing employment retention bonuses as part of the Transitional Jobs Reentry Demonstration project.¹²

• Work experience is not enough. As noted earlier, most subsidized employment programs provide some ancillary supports to address personal or situational barriers that participants face (for example, substance abuse, heavy child support obligations, child care needs, unstable housing, etc.). However, in most cases, these services are not closely integrated with the subsidized employment program and must be accessed by referrals. It is possible that, for some participants, these issues overwhelm the benefits of work experience and hinder long-term employment outcomes.

• The programs are too short. Perhaps participants would learn more and benefit more if they were able to stay in subsidized jobs for longer periods, or perhaps the programs should not be thought of as sequential. In other words, rather than assuming that participants will move in lockstep from subsidized jobs to unsubsidized jobs, programs could encourage participants to come back to subsidized jobs, at least temporarily, if they lose an unsubsidized job.

• The programs don’t focus enough on the “transition.” Perhaps clients learn valuable soft skills, but the programs do not capitalize on this fact in their job placement and post-placement services. Implementation studies have found that there is tendency for program staff to concentrate on the complex task of creating and/or managing worksites, which potentially diverts attention away from the “back end” of the program.

The recent transitional jobs evaluations have provided hints as to which of these hypotheses seem most relevant, but there is much more to learn. Building on these recent studies, a key goal of the ETJD and STED projects is to integrate process and impact data to gain more detailed insight into why programs do or do not achieve their long term goals.

Framework for the STED and ETJD Evaluations

Exhibit 1.1 illustrates the general framework for the ETJD and STED evaluations in any site. Box A represents the potential target population for the program being tested. The characteristics of the evaluation’s research sample (Box B) are shaped by the program’s recruitment or referral and screening procedures. It will be important for the study to understand how the research sample fits into the broader population from which it is drawn, and how the programs sought to identify who is appropriate for a subsidized job.

Sample members are randomly assigned to a program group that is offered (or, in some contexts, required to participate in) a program that includes subsidized employment, or to a control group that does not participate in the program being tested. Box C shows the key dimensions of the ETJD/STED programs that were measured in the implementation study: the model (including both characteristics of the work experience component and other services that are provided), its implementation, and the dosage of services that sample members receive. Control group members are provided a resource list informing them of other community services they may access. They will have access to all services except those directly provided through the ETJD or STED project grants. Box D depicts the services that the control group will receive in the community. The “I” indicates that the study will compare the type and amount of services received by the program and control groups in order to understand the treatment difference.

Initially, the evaluation will measure short-term outcomes on employment and earnings, which will likely be driven by the subsidized jobs (see Boxes E and F). The study will also measure key indirect outcomes such as welfare receipt, child support, recidivism, or father-child relationships, with the specifics depending on the target population. Over the longer-term, the study will assess whether the program leads to gains in unsubsidized employment.

Finally, the long box at the far right represents the context in which the program operates. Elements of the context – the labor market, characteristics of the system in which the program operates, the service environment – will shape the target population, the services both groups receive, and the outcomes. For example, if a program operates within the TANF system, then the counterfactual (i.e., control group experience) will be profoundly shaped by the jurisdiction’s overall approach to TANF employment services. Thus, the implementation study will examine the local context in each site and we will also collect data on local labor market conditions.

The new STED site will follow a similar evaluation framework. The target population will include individuals who: earned less than $30,000 in the year prior to entering the study; have a Social Security Number; are between the ages of 21 and 64; plan to file income tax returns as single without dependent children; and do not receive Supplemental Security Income or Social Security Disability Insurance. The study will recruit 4,002 individuals starting in late 2015, pending OMB approval, and randomly assign half of them to a treatment or a control group.

A.1.2. Overview of the STED and ETJD Evaluations

As noted, the STED and ETJD projects have complementary goals and are focusing on related program models and target populations. The ETJD project targets low-income noncustodial parents and ex-offenders. At this point, seven ETJD grantees have been selected and funded by ETA and these sites began to enroll participants in late 2011. Two of these seven sites are being evaluated as part of the STED evaluation. In addition, ACF has recruited an additional six sites for the STED evaluation, resulting in a total of 13 sites between the two evaluations. ACF received clearance from OMB to have discussions with states as part of the Subsidizing Employment Opportunities for Low-Income Families: A Review of State Employment Programs created through the TANF Emergency Fund (OMB Control Number 0970-0384). Three of the STED sites target current, former, or potential TANF recipients; two of the STED sites target disadvantaged youth and young adults; and the new STED site targets low-income single individuals without dependent children. Each of the sites in the ETJD study have enrolled approximately 1,000 study participants. Two of the STED sites have enrolled approximately 2,700 study participants; one has enrolled 820 participants; one is projected to enroll1,000 participants; one is projected to enroll 400 participants; and the new STED site is projected to enroll 4,002 participants. Overall, the total participant count in the STED and ETJD studies combined is expected to be 18, 222. See A.12 for more information about adjustments to burden estimates compared to the previous estimates.

The STED and ETJD evaluations are both being led by MDRC, a nonprofit organization which has contracts with both ACF and ETA. The evaluations include the following components:

Implementation and process study: The goal of this component is to understand how the programs operate, who they serve, the context in which they operate, and the dosage of services that sample members receive.

Impact study: The goal of this component is to assess how the STED and ETJD programs affect employment, earnings, and other key outcomes such as recidivism rates, child support payments, well-being, and family relationships.

Cost effectiveness study: This component will estimate the net cost per person of the STED and ETJD programs and compare those figures to the financial benefits that the program generate (for example, reduced incarceration or public assistance payments).

The two studies are collecting four categories of data: 1) baseline data, 2) implementation and process data (some of which will support the cost effectiveness study), 3) surveys of research sample members, and 4) administrative records.

Baseline data for the two evaluations is collected through the Contact Form (Appendix C – previously approved) and the Baseline Information Form (Appendix D – previously approved). In addition, sample members will be asked to provide updated contact information at several points following random assignment. Tracking letters requesting this updated information are included in Appendix E (previously approved). This revision to the information collection request seeks approval for the Contact Form and Baseline Information Form for the new STED site, which are presented in Appendix X and Appendix Y, respectively.

Implementation and process study materials include four short questionnaires to be completed by program staff, participants, worksite supervisors, and employers; interview guides for program staff members and administrators, employers, and referral partners; a discussion guide for participant focus groups; and time-use worksheet for program staff (which will also inform the cost study). These materials are included in Appendices F through I (all previously approved) and are described in detail below.

Follow-up data will be collected through three surveys to be administered to study participants 6¹³,12, and 30 months following random assignment. The projects will attempt to survey most¹⁴ STED participants at 6, 12, and 30 months and all ETJD participants at 12 and 30 months. The 6-,12-, and 30-month follow-up surveys were reviewed and approved through previous information collection requests.

The new STED site will not collect implementation and process data or administer surveys to participants. It will collect administrative records data on employment and earnings, benefit receipt, and child support.

A.2. How, by Whom, and for What Purpose the Information is to be Used

How the Information will be used?

The findings from both studies will be used to inform the Federal government, states operating subsidized and transitional employment programs and expanded EITC programs for individuals without dependent children, program administrators, employers, and other stakeholders about the feasibility and the effectiveness of subsidized and transitional employment programs and expanded EITC programs for individuals without dependent children in helping vulnerable populations secure unsubsidized jobs in the labor market and achieve self-sufficiency.

Who Will Collect the Information

MDRC, the contractor for both studies, will collect data through face-to-face interviews with key informants, short questionnaires, site visits (including observations of program activities and case file reviews), and surveys. Program data and other existing data will be analyzed.

Baseline Information

Informed consent, contact information, and baseline information (Appendices B-D and W-Y) is collected from all study participants following determination of program eligibility, but prior to random assignment. These data will provide general demographic information about study participants, help to ensure the comparability of the program and control groups, allow for subgroup analyses, and facilitate contact for subsequent follow-up surveys. Prior to enrolling in the study, staff in the new STED site will discuss the program and evaluation with potential participants. Appendix Z includes staff talking points for introducing the study and guiding a participant through enrollment.

Although there is no follow-up survey currently planned for the new STED site, the contact information form will help to maintain contact with individuals assigned to the program group, who are eligible for the expanded credit. The contact information for both groups will facilitate contact for a later survey, should such a survey be added to the evaluation.

Procedures for baseline data collection will differ slightly between the ETJD and STED projects. Notably, the seven ETJD sites (including the two that have joined the STED project) have already enrolled participants, obtained informed consent, and collected baseline and contact data. Collection of baseline data for the ETJD sites received OMB clearance on September 9, 2011 (OMB Control Number 1205-0485). Baseline data were collected via a Management Information System (MIS) developed and managed by ETA. The baseline data that are being collected for the six STED sites are similar to those being collected in ETJD sites. Where there are differences, we have described them below.

Informed Consent Form: Each of the six STED sites has individualized informed consent forms (Appendix B); the two STED sites that target youths and young adults have a separate informed consent form for participants that are minors at the time of study enrollment; the new, expanded EITC STED site has separate informed consent forms that explain the study and allow individuals to consent to have their tax data shared for research purposes and for the purposes of calculating their EITC amount (Appendix W). These will ensure that participants: 1) understand the STED evaluation, as well as their role and rights within the study; and 2) provide their consent to participate.

To ensure that all study participants receive a clear, consistent explanation of the project, the evaluation team will train program staff on how to introduce and discuss the goals and design of the project, the random assignment process and data collection efforts. All staff will emphasize: 1) that participation in the study is voluntary; and 2) that strict rules are in place to protect sample members’ privacy.

Contact Information Form: Contact information in the five STED sites was or will be collected prior to random assignment. In addition to their own name, address, and phone numbers, participants will be asked to provide information about three additional individuals who are likely to be in future contact with them and can assist the research team to locate them for follow-up surveys. This form is attached in Appendix C. The contact information form for the new STED site is shown in Appendix X.

Baseline Information Form: The baseline form (Appendix D) collects basic identifying information about sample members in the five STED sites, including social security number and date of birth. It also includes demographic items such as race and ethnicity, primary language spoken, and housing and marital status. Finally, the form also includes questions about participants’ education and employment history and any involvement with public assistance. These data will be used by the evaluation team to ensure the comparability of program and control groups, to facilitate contact for follow-up surveys, and to inform subgroup analyses in the impact study. Baseline forms will be administered to individual sample members by program staff members in one-on-one or group settings, with program staff members available in all cases to answer any questions.

As noted above, in the seven ETJD sites (including the two sites that have joined STED), baseline data that have already received OMB clearance (see above) are being collected via a Management Information System. In addition to the data that have already received OMB clearance, the STED baseline information form contains a few questions about public assistance receipt which do not appear on the forms administered in ETJD sites. We have also removed some questions about criminal history and child support status which appear on the baseline information forms being administered at the seven ETJD sites. These changes were made because we expect different populations of participants in the two projects, with more TANF and public assistance recipients in STED programs and more ex-offenders and non-custodial parents in ETJD programs.

The baseline from for the new STED site (Appendix Y) is very similar to the approved STED form. However, the following items were added to reflect the different target population and the fact that the Paycheck Plus bonus is obtained by filing income tax returns:

• self-employment status - relevant because the process and benefit of filing income tax returns are different for the self-employed;

• date of most recent incarceration – relevant in order to determining subgroups that might differ in their response to the bonus;

• physical and emotional well-being – these are measured by several items, including the presence of physical or mental health limitations, self-rated health status, and overall happiness. These questions are relevant for determining subgroups more or less likely to take advantage of a work bonus;

• knowledge of the EITC – relevant for likely understanding of program bonus; and

• whether filed income tax return last year – relevant for experience with filing income tax returns.

These changes were made to reflect the different target population, many of whom may be former prisoners and non-custodial parents, and the fact that the bonus is obtained by filing income tax returns.

Tracking Letters.  Because sample members are likely to be highly mobile and difficult to locate, the evaluation team will request updated contact information by mail from all sample members at various points following random assignment. Appendix E includes samples of the letters that will be sent to sample members. Each of these letters requests that participants call a toll-free number or return a form providing updated contact information for the respondent and—in some cases—up to three other contacts. Study participants in the new STED site will not receive tracking letters.

Sample members in the ETJD evaluation will be sent tracking letters at six points following random assignment (at 3, 9, 16, 20, 24, and 28 months).  These letters will remind participants of their participation in the ETJD study and will request that they notify the survey team if any personal contact information has changed since their intake into the study. The first of these letters will contain a $2 bill and all subsequent letters promise participants $5 for responding to the letter with updated contact information.¹⁵ ETJD participants will also be sent letters prior to the administration of the 12 and 30-month surveys notifying them that they will soon be contacted by phone to take part in a survey.  

STED participants will be sent two different kinds of letters following random assignment.  Each STED participant will be sent a letter of greeting on each birthday that occurs between random assignment and the administration of the 30-month survey. STED participants will also be sent tracking letters at 18 and 24 months post-random assignment. This letter requests that the participant update his/her contact information if it has changed. A few days prior to the administration of the 6, 12, and 30-month surveys, STED participants will also receive a letter reminding them of their participation in the study and notifying them that they will be contacted to participate in a survey shortly. Each of these notification letters will be nearly identical in content, but will be updated according to which survey is to be administered.

Implementation and Process Data

Implementation and process data will be collected during visits to all selected ETJD and STED sites approximately one year and two years after the beginning of program operations. While on site, evaluation staff will conduct interviews with program staff members and administrators, interviews with employers and referral partners, review participant cases, and observe program activities. Attached with this package in Appendix J are the protocols that will be used for interviews with staff members, employers, and referral partners. Discussion guides for focus groups with participants are attached in Appendix K. The form that evaluation staff will use to guide their observations of worksite activities is also attached with this package as Appendix L. Implementation and process data will not be collected for the new STED site.

In addition, the evaluation teams intend to administer four questionnaires at different points during the life of the project: a program staff questionnaire; a worksite supervisor questionnaire; a participant questionnaire (to be administered to participants actively engaged in subsidized employment); and an employer questionnaire. These questionaires will obtain quantifiable data on dimensions of program implementation that will facilitate cross-site comparisons and help to explain any differences in program impacts across sites. All of these questionnaires are described below and attached in Appendix F through I.

Finally, the evaluation team will ask program staff members to record the time they spend on a range of program activities. This information will inform the implementation research by providing information on what percentage of time staff members are spending on different activities and in direct engagement with program participants (through such activities as case management, referrals, job counseling, job development, etc.). This information will supplement interviews and on-site observations by providing information on time-use that is directly comparable across sites. It will also inform the analysis of program costs by allowing the team to allocate the percentage of program budgets that is devoted to the activities measured. This worksheet is attached in Appendix M.

All questions have also been extensively reviewed by the evaluation staff, other senior MDRC staff, staff members at ACF and ETA, and members of the STED Technical Working Group (TWG). A list of TWG members is attached as Appendix Q.

Data collection activities are described below.

Participant Implementation Questionnaire. The participant questionnaire (Appendix G) will supplement responses to the 6- and 12-month surveys by providing information about participants’ experience on the job while they are still engaged in subsidized employment. The questionnaire will be administered three times over the course of the evaluation to all participants who are engaged in subsidized employment at the selected point in time. We expect to administer these questionnaires at approximately 6-9 month intervals so that no individual participant will be asked to complete the form more than once. These questionnaires take approximately 10 minutes to complete; they are administered with the help of program or research staff. In order to maximize response rates, and out of consideration for the time participants spend completing the questionnaire, we provide a nominal monetary incentive for completing the participant implementation questionnaire. This level was previously approved by OMB and is provided in the form of a gift card and will not exceed $10.

The participant questionnaire includes approximately 25 questions on the following topics:

1. Skill development. Transitional and subsidized jobs programs aim to increase participants’ employability, partly through the acquisition of “soft skills” – that is, general employability skills such as showing up on time and working well with others. Some programs teach these skills directly in pre-employment training, others hope that experiences on the job either teach or advance these skills. Hard skills – such as computer or construction skills -- are provided in some programs. The Participant Questionnaire attempts to ascertain whether participants believe they are acquiring soft and/or hard skills through their subsidized jobs.

2. Supervision. Supervision and on-the-job coaching are important mechanisms for correcting problems and improving participants’ employability. The Participant Questionnaire contains questions about interactions with supervisors to understand how much time supervisors spend with participants and the content of their interactions.

3. Peer support. Interactions with peers are also thought to be important in changing work behaviors and increasing contacts to help participants get and retain subsequent jobs. The Participant Questionnaire contains questions about support from peers.

The questions for the participant questionnaire were drawn primarily from surveys that were administered to participants in the JOBSTART evaluation and participants in the CEO transitional jobs program for ex-offenders. Those questions that were not drawn from these sources were developed internally by MDRC and its partners. They were developed to provide greater information about subsidized employment participants’ own perceptions of the benefit of their experience and the specific skills and lessons they are taking from it. Many members of the evaluation team have deep knowledge and experience related to transitional jobs and employment programs for the populations in the STED and ETJD evaluations, but there is not a rich source of information providing self-reported experiences of participants while they are going through programs. This questionnaire will fill some of these gaps in existing knowledge and also allow us to compare the experience of participants across sites

Participant focus groups. Focus groups with participants are being conducted twice in each site. These group interviews are being conducted with about 10 individuals currently participating in the program. These participants will be selected based on who is available to participate in the focus group at the time we are conducting it. In our experience, fewer participants show up than we expect. Group interviews collect important information on how prepared participants felt when starting their subsidized job, views about their subsidized jobs such as what skills they think they are learning, frequency and nature of supervision, and the support they receive from program staff and/or peers. A protocol for these discussions is included in Appendix K.

Program Staff Implementation Questionnaire The program staff questionnaire (Appendix F) will supplement semi-structured interviews with program staff. The questionnaire is administered to key program staff who are involved in program administration or who work directly with participants (such as job developers and case managers). Each staff person will complete the questionnaire once during implementation visits to STED and ETJD site locations; it requires approximately 10 minutes to complete.

The staff questionnaire is not intended to duplicate the information that was collected via the semi-structured interviews discussed below. Instead the survey will focus on two aspects of implementation: 1) job development and 2) general organizational issues and work environment.

1. Job Development/Placement. Questions focus on job development and placement in the transitional or subsidized job as well as program staff communications with the transitional or subsidized job employers. In addition, since a critical component of these programs is the connection into unsubsidized employment, there are questions about what methods program staff use to place participants into unsubsidized employment as well as how they talk to participants about making the transition.

2. Work Environment/Organizational Capacity. Questions cover issues such as whether staff get clear direction, regular feedback from supervisors, and staff morale levels. These “generic” organizational factors may be critical in explaining differences in implementation and impacts across sites.

The questions for the Program Staff Questionnaire were drawn primarily from two sources: 1) the Chicago Neighborhood JobStart Staff Survey evaluated by the Social IMPACT Research Center and 2) the Minnesota Family Investment Program Job Counselor Survey, which was administered by MDRC and received OMB clearance. Those questions that were not drawn from these sources were developed internally by MDRC and its partners/subcontractors—MEF Associates, Branch Associates, Abt Associates, and DIR, Inc.

This questionnaire was developed to provide information about the nature of job development and whether job placement for the subsidized jobs emphasizes placements that match participants’ skills and interests or meets employers’ needs and how program staff balance these considerations while achieving placement goals. In addition, it is hypothesized that work environment is positively correlated with staff members’ abilities to perform job duties and these questions will ensure uniform data on organizational capacity and the work environment across the sites. This questionnaire will allow the team to more systematically compare the nature of job development and the job developer/case managers’ work environments across sites and provide contextual information for understanding any variation that may occur in outcomes and impacts.

Worksite Supervisor Implementation Questionnaire. The worksite supervisor questionnaire (Appendix H) provides information about how supervisors view their role in working with subsidized employees and the experience that supervisors have working with participants in the evaluation. This questionnaire is administered to the direct supervisors of program participants while evaluation staff are on-site for implementation visits. It is administered twice over the course of the evaluation, although we only expect each supervisor to complete it once. MDRC will keep track of supervisor names to ensure that each supervisor completes it only once; however, supervisors are not required to write her or his name on the questionnaire, if s/he wants to remain anonymous. The questionnaire is administered on paper and takes approximately 10 minutes for supervisors to complete. In order to maximize response rates, and out of consideration for the time worksite supervisors spend completing the questionnaire, we provide a nominal monetary incentive for completing the worksite supervisor implementation questionnaire. This level was previously approved by OMB and is provided in the form of a gift card and will not exceed $10.  

The worksite supervisor questionnaire captures some of the same topics as the participant questionnaire – but from the supervisors’ perspective. In some cases, these supervisors are employees of the program site. In other cases, they are staff of private employers. The goal of the worksite supervisor questionnaire is to obtain data on the nature of supervision and how program participants compare with other, unsubsidized workers in terms of productivity and supervision. Key topics include:

Participant Performance. Work site supervisors have the most direct contact with participants on the job, and thus are well-suited to provide information about participants’ performance on the job (relative to others in the same position) and the disciplinary actions they take.

Skill development. Prior evaluations have demonstrated that it can be challenging to ensure that work site experiences provide opportunities for participants to build soft skills. Work site supervisors – especially in private businesses – may be more concerned with getting the work done than coaching participants. This survey captures the extent to which supervisors spend time coaching participants.

Communication with Program Staff. Also important in program implementation is the communication between work sites and case managers and this survey asks about communication and feedback given to program staff.

Many of the questions for worksite supervisors were developed internally by MDRC and its partners based on prior experience evaluating transitional jobs and employment programs. Some of the questions have been drawn from surveys that were administered in the JOBSTART evaluation and in the evaluation of the CEO transitional jobs program for ex-offenders. Like the participant questionnaire, this questionnaire allows the team to systematically compare the experience of supervisors across sites and to better understand what kinds of approaches may have the most value for subsidized employees and the jobs in which they are working. It provides more contextual information for understanding any variation that may occur in outcomes and impacts across the sites.

Employer Implementation Questionnaire. The employer questionnaire (Appendix I) collects basic information about employers who take on subsidized workers. It is administered twice over the course of the evaluation, although we only expect each employer to complete it once. MDRC will keep track of employer names to ensure that each employer completes it only once; however, the employer is not required to write her or his name on the questionnaire, if s/he wants to remain anonymous. It will be administered on paper with the help of program staff.

Questionnaires are sent to the contact person identified by program staff and are administered in person as available. The individual who made the decision to take on subsidized workers and/or who handled the hiring is asked to complete the questionnaire. It collects basic information about the respondent’s role to ensure that the correct person is completing the questionnaire. It takes approximately 10 minutes to complete. In order to maximize response rates, and out of consideration for the time employers spend completing the questionnaire, we provide a nominal monetary incentive for completing the employer implementation questionnaire. This level was previously approved by OMB and is provided in the form of a gift card and will not exceed $10.

Since work experience is typically the heart of subsidized employment programs, it is critical to understand, in some detail, the characteristics of participating employers and their experiences in the program. The questionnaire will collect information on employers’ characteristics such as the number of employees, type of business, and financial health. The questionnaire also captures experiences in the program such workers’ productivity and changes in the firm’s financial health.

Questions for the employer questionnaire have been drawn from a number of sources. Certain questions were developed internally by MDRC and its partners but the majority of questions have been drawn from surveys that have been administered to employers as a part of previous evaluations and employment projects. These sources include the Chicago Neighborhood JOBSTART Employer survey; Bridging the Opportunity Gap: The Massachusetts Department of Youth Services Career Readiness Vocational and Employability Re-entry Initiative; the On-the-Job Training Survey conducted by the Center for Workforce Information and Analysis at the Pennsylvania Department of Labor and Industry; Subsidizing On-the-Job Training: An Analysis of a National Survey of Employers conducted by John Bishop for ETA; Entry-Level Employers in Colorado: Results from a Survey of 25 Employers by Burt Barnow; and Work Opportunity Tax Credit, a 2001 GAO report presented to the House Committee on Ways and Means.

Like the participant questionnaire, the employer questionnaire allows the team to systematically compare across sites and to better understand what kinds of approaches may have the most value for subsidized employees and the jobs in which they are working. It provides more contextual information for understanding any variation that may occur in outcomes and impacts across the sites. Specifically, the data will be used to determine whether firms tend to place subsidized workers in vacant job slots or create new positions. The data will also be used to explore whether certain features of participating employers are correlated with variations in outcomes across the sites.

Case file reviews. Case reviews are an effective way to obtain detailed, quantifiable data on certain implementation practices. The reviews provide a systematic vehicle to probe beyond the data that are typically included in a management information system. While on site, evaluation staff members will document the extent of interaction between the job developer and the client, the extent to which job development was tailored to the individual client, the number of specific job referrals that the client received, the extent of contact between job developers and employers, and so on. Using job development/placement as an example, we are using program MIS data to select a random sample of 10-20 cases that were placed in jobs and another sample of 10-20 cases that received job development services but were not placed. If case notes and other data are maintained electronically, we will review the cases before our field visit to supplement the on-site review of documents. We will to gather information for approximately 20-40 participants in each site during each implementation site visit. We do not expect our case file reviews to place a burden on program staff.

Interviews with program staff, employers and referral partners. Semi-structured interviews with program staff (project directors, case managers, job developers, and job coaches), employers, and referral partners are conducted twice, during visits to the participating sites. The interviews are used to obtain basic information about the program model and its implementation as well as to understand the organizational capacity of the primary agency and any partners and how they staff the program. These interviews cover basic aspects of the program including the recruitment and enrollment process, assessing participants’ skills and needs, the subsidized jobs (type of work, supervision, length), pre-employment training, skills training, case management, job development, and post-placement services. These interviews will also be the source for contextual information such as other initiatives and features of the local economy.

The first site visit was conducted about one year after the start of random assignment and included questions about the background of the program model, recruitment and screening, program goals, target population, subsidy details and mechanics. Interviews took about one hour to complete, and were conducted either one-on-one or in a group.

The second site visit will be conducted about 12 months later. The purpose of the second interview will be to elicit perceptions of how well plans have been implemented and reasons for departures from what was planned. Again, interviews will take about one hour to complete, and will be conducted either one-on-one or in a group.

Appendix J includes interview guides for program staff, directors, employers, and referral partners (including parole officers). Since each site will have different a staffing structure, not all the questions listed in these protocols will be asked of each respondent we interview; rather, the protocols will serve as guides to evaluation staff conducting interviews.

Worksite observations. It is critical to understand, in some detail, what goes on at subsidized worksites. To address this issue, the research team will supplement interview and focus group data with direct worksite observations.

Our exact procedures for observing worksite activities varies depending on whether subsidized workers are spread among many worksites or consolidated in a few. In the former case, we will visit and observe a sample of worksites while in the latter we will visit the single worksite. In either case, a member of the research team will spend approximately one to two hours observing the subsidized worksite and completing an observation check list (Appendix L) covering cover issues such as: 1) what kind of work STED and ETJD participants are doing; 2) the extent and nature of interactions between the participants and co-workers or customers; and 3) the extent and nature of interaction between participants and supervisors. We do not expect our observation of worksite activities to place any burden on site staff, supervisors, or participants.

Program Staff Time-use Worksheet. The staff time-use worksheet (Appendix M) serves both the implementation and cost studies. It is administered to all program staff from state and local agencies and program provider organizations that are participating in the STED and ETJD evaluations. The worksheet captures all of the relevant activities that any of the programs might provide, although each program will likely record time in a limited number of categories based on the program intervention. Each program staff person is asked to keep track of their time for a two-week period, to be determined by the program senior staff and the research team.

The staff time worksheet is being used for the implementation study to describe how staff members spend their time. It allows the study team to describe the emphasis placed by the program on each program activity and allow for comparisons to be made across sites. For example, it allows the team to compare program case management and job development practices across sites.

The time-use worksheet is also being used for the cost study to allocate the staff salaries and overhead costs incurred by the program to each program activity based on the percent of time that staff spent on each activity. Using the cost information, time-use worksheet, and participation data, the research team will be able to estimate the average cost of providing each STED/ETJD service to participants.

Cost Data Collection Protocol: Appendix N includes a protocol administered with a program fiscal or accounting staff member while on site for implementation visits. Before we administer it to program staff, we will have already received complete financial expenditure reports from the program. For example, these reports include data on staff salaries, rent, overhead, supplies, etc. As such, the protocol is used as a guide to supplement these reports. In addition, we will use the program staff time-use worksheet in conducting our cost analysis. The burden for this protocol is under program staff interviews, as it will be used during the site visits. We included one additional program staff from each site to the burden estimate.

Survey Data Collection

ETJD and STED sample members will be surveyed at different points following random assignment. OMB has reviewed and approved 6- and 12- month follow-up surveys. This information collection request is specific to an additional follow-up survey at thirty months. Sample members in four of the five STED sites will be contacted and asked to complete follow-up interviews approximately 6, 12, and 30 months after they are randomly assigned; sample members in the fifth STED site will be contacted and asked to complete the 6-month follow-up 6 to 9 months after they are randomly assignment. Sample members in the five non-STED ETJD sites will be contacted 12, and 30 months after they are randomly assigned. The survey data will contribute substantially to the ability of the STED/ETJD evaluations to measure the effectiveness of different strategies to increase employment among hard-to-employ populations with the long-term goal of making individuals and families better off. Specifically, data collected will enable us to determine whether or not the resources allocated to the various subsidized and transitional employment strategies did, in fact, lead to increased employment, higher incomes, and improved wellbeing. The anticipated response rate for each survey is 80%.

Although administrative records data will play a critical role in the evaluation, they leave some important gaps in knowledge about a range of outcomes that are very relevant to the study. The surveys will yield important data not available through administrative records, such as information on the characteristics of jobs held during the follow-up period (e.g., wage rates, hours worked, and fringe benefits), participation in employment-related services, parental involvement, and material and financial hardships. The surveys will also provide information on the sources of program and control members’ income—including disability payments, Earned Income Tax Credit (EITC), and alimony payments—that are unavailable from the administrative data. Furthermore, the survey is the only source of information on earnings and other income received by other members of respondents’ households. Surveys will not be fielded in the new Paycheck Plus site, given that the primary outcome of interest will be employment and earnings obtained from earnings data.

The surveys will also provide important information for the study’s cost analysis, by detailing the types of activities and work supports the individual has participated in or received six months and one year prior to the survey interview. This information will be helpful for establishing the net cost of the program interventions. While program records may be a good source of cost data for the program group, there is no way to collect similar information on the control group, since in most cases little information is available on them.

There are two versions of the 6 and 12-month surveys: one for the sites targeting young adults and youth and one for sites target adult populations (former/current/potential TANF recipients, non-custodial parents, and ex-offenders). The young adult version of the surveys includes more items related to education/training and items related to expectations and planning; as these are areas emphasized in the interventions at the two STED youth/young adult sites. All participants will be administered the same survey at 30 months. All three surveys are discussed in more detail below, including further details of the differences between the young adult and adult versions of the 6 and 12-month instruments.

6-Month Survey. The 6-month survey (Appendix O) is a 30 minute survey which is mostly focused on measuring the in-program effects of STED programs during a time period that has been understudied in past evaluations of these programs. In particular, the survey is administered while many pr0ogram group members are participating in subsidized employment, a period when there is likely to be a very large impact on employment (i.e., the program group will be much more likely to be employed owing to the subsidized jobs).¹⁶

This timing provides an opportunity to go beyond past surveys for evaluations of subsidized and transitional job programs, which have mostly focused on the economic benefits of working. It may be that employment confers a range of nonfinancial benefits. Evidence suggests that unemployment increases stress, anxiety and depression. These can lead to additional incidence of disease, reduced job search activities, damage to social and familial relationships, and social withdrawal. Work provides income and potentially income stability. These can reduce stress and anxiety over basic survival issues like paying bills and providing food to one’s family and thus remove the triggers for a host of problems including poor physical health and strain on relationships. However, the experience of work seems to matter as well. That is, work may do more than simply relieve the hardship of unemployment.

Work ties into conceptions of self-worth and self-efficacy. Success at work demonstrates competence to oneself and to one’s family and friends. Furthermore, individuals may benefit from simply “getting out of the house,” and interacting with others. Research on depression has repeatedly emphasized the importance of regular social contact in promoting psychological well-being. Work may also strengthen the social support available to an individual. Interaction with co-workers might foster positive peer effects or generate connections that widen the social networks that participants often use to find work. For ex-offender populations, for example, exposure to and integration into the larger population may provide positive peer “pressure” and possible mentoring opportunities that could reduce recidivism to prison.

The 6-month survey provides a very rare opportunity to examine some of these issues in the context of a large experimentally-induced impact on employment. Typically, research on the spillover effects of employment relies on correlational data and is subject to selection bias.

What follows is a brief description of each section of the 6-month survey:

• Employment and Service Participation. The proposed survey begins by trying to measure the current employment and service receipt status of study participants. These findings will help the research team to better understand the pattern of findings on other outcomes. For example, if there are no effects on employment, this would help explain a lack of effects on the nonfinancial benefits that are expected to accrue from employment. For the young adult version of the survey, the employment section was revised to include participation in internships, paid and un-paid, as this form of employment is more typical for the youth/young adult population. The sequence of the questions in this section was also revised to capture engagement in education and training, as this type of productive activity is more likely for the youth/young adult population.

• Physical and Mental Health. The survey then proceeds to ask a series of questions (derived from national surveys) about physical and mental health. Job loss and job experiences can be associated with a variety of mental health effects including stress, anxiety, and depression. Poor mental health may manifest somatically in a variety of physical health problems ranging from aches/pains to serious heart conditions so some measures of general physical well-being are included. The young adult version of the surveys uses the same measures as the adult version.

• Efficacy and Locus of Control. A series of psychological measures are collected in the areas of depression, locus of control, and efficacy. General Self-Efficacy is related to self-confidence and motivation/effort and thus a possible intermediate outcome of interest. This could presumably be associated with better work performance (and thus retention) or more rigorous job search. Locus of control captures respondent beliefs about how much of their lives are directly under their control as opposed to the result of luck, fate or manipulation by other individuals. While related, perceptions of self-efficacy and locus of control are usually considered distinct and thus measured separately. The young adult version of the survey includes additional items related to these topics as these domains are of particular interest for this population. The adult survey includes items related to job search and work self efficacy: two domains that are especially relevant to this study. The young adult version includes items related to career orientation, as this population is not expected to have sufficient labor market experience to have formed meaningful perceptions of their job-search and work self efficacy.

• Social Isolation and Social Support. Social isolation and poor social support limit individuals in several ways. Most individuals find their jobs through social networks and may do so passively (stumbling upon job leads/information/offers). Poorly connected individuals have fewer opportunities for this to occur, less access to novel information, and fewer friends/family/acquaintances to query. Social networks also provide needed financial and non-financial support (e.g. baby-sitting, cooking, and transportation) that can help individual weather difficult situations and facilitate job participation. For the young adults, this survey also assesses the presence of mentors, roles models and other influential advisors.

• Social Network Roster and Relationship Origin. Rosters allow more direct measurement of social support. The questions here identify the members of the social networks of respondents and how the network members are known to the respondent. This allows assessment of whether program group members (1) have larger networks (i.e., list more relationships) and (2) have different relational compositions (i.e., list more work or program related relationships). The adult and young adult versions of the surveys do not differ for this topic area.

12-month Survey. The 12-month survey (Appendix P) collects data on study participants’ receipt of services and attainment of education credentials, labor market status, material hardship, household income, criminal justice, self-sufficiency and family engagement, including, child support payments and parent-child contact. The 12-month survey takes 45 minutes to administer.

The 12-month survey is comprised of several modules. Most of these modules have two purposes: (1) to provide a systematic description of key outcomes and; and (2) to measure the differences in employment, wage progression, income, and other outcomes between the program groups and a similar group of respondents who were not eligible for the programs. What follows is a summary of the proposed modules in the STED and ETJD 12-month survey:

• Participation in Employment-Related and Education Activities: This modules measures the extent of participation in a range of activities (including job search and education and training), for both the program and control groups. It is the key source of data for measuring the treatment difference that will drive the later impacts on longer-term outcomes. The young adult version of the survey emphasizes engagement in education and training as this type of participation is more relevant to the population.

• Employment: This module includes an employment history since random assignment and measures a variety of characteristics of each job held. Employment outcomes will be analyzed separately depending on whether employment is in subsidized or unsubsidized jobs. Both versions of the survey use the same employment module, except that the young adult version includes internships as a form of employment for the reasons noted above.

• Marriage, Household Composition, and Material Hardship: This module measures housing and marital status and family composition. It also measures income, the primary income sources (such as child support, Supplemental Security Income, and EITC) during a one-month period at the time of the interview. Finally, this section measures levels of material and financial hardship. The young adult version of the survey includes additional items related to housing, particularly changes in housing status, as stability in living circumstances is more critical for this population. In addition, the young adult version of the survey focuses on individual income and material hardship while the adult version includes both individual and household income/hardship; this is due to the expected greater prevalence of communal living situations for the young adult population (i.e., they are more likely to be living with room-mates and/or still in the family home).

• Health Coverage: This module measures the extent to which respondents have health coverage, funded by employers or other private sources, or funded by government programs like Medicaid. For the young adult version of the survey, this also includes being covered by their parent or guardian’s health insurance.

• Health and well-being: This module measures the extent to which respondents or their family members have any key health problems. This section includes some of the psychological scales from the 6-month survey. The young adult version of the survey includes greater focus on psychological well-being, including substance abuse and mental health treatment. For the young adults, most of the items from the earlier survey are replicated at 12 months as they are key outcomes for these populations.

• Criminal justice module: This module measures outcomes that are not measured, or not measured well by administrative records including parole, probation violations and recidivism. This section is administered to all participants in the sites targeting ex-offenders, and only those with a criminal record in the noncustodial parent (NCP), and TANF sites. Sample members at both of the young adult sites are administered this module as criminal justice involvement is considered of high importance for this population.

• Custodial parents module: This section is for the sites targeting TANF recipients. The focus is on the cost of child care and child care barriers that participants might confront which might affect their program participation or employment. The young adult survey includes some items on these topics but is less extensive as fewer sample members are expected to be custodial parents.

• Noncustodial parenting module: These questions will be asked of all participants in the NCP sites and NCPs in the Ex-Offender sites. The focus is on child support payments and parental involvement. The young adult survey also includes some items related to this topic but the coverage is less extensive as fewer sample members are expected to be non-custodial parents.

30-month Survey. The 30-month survey (Appendix U) will collect data on study participants’ education credentials, labor market status, material hardship and income, health and well-being, social support and networks, criminal justice, self-sufficiency and family engagement, including child support payments and parent-child contact. We expect the 30-month survey to take 30 minutes to administer.

The 30-month survey is comprised of several modules. Most of these modules have two purposes: (1) to provide a systematic description of key outcomes and; and (2) to measure the differences in employment, wage progression, income, and other outcomes between the program groups and a similar group of respondents who were not eligible for the programs. Most of the items in the 30-month survey are taken from the earlier follow-up surveys. The changes are detailed in Appendix T (Overview of the 30-month Follow-Up Survey). What follows is a summary of the proposed modules in the STED and ETJD 30-month survey:

• Education Credentials: This module measures respondents’ current education attainment, including occupational certifications and licenses. Also included are items concerning current educational activities. While neither program is specifically directed towards increasing educational attainment, both aim to produce increased attachment to the labor force which could result in increased investments in education and training.

• Employment: This module includes an employment history since random assignment and measures a variety of characteristics of each job held¹⁷. For respondents that are currently or recently employed, this module also includes items concerning job quality, including strenuous work conditions, job satisfaction, and advancement opportunities. Employment outcomes will be analyzed separately depending on whether employment is in subsidized or unsubsidized jobs.

• Living Conditions and Well-Being: This module measures housing, marital status and family composition, income, material and financial hardship, and physical and mental health. Also included are items related to health coverage for respondents, spouses/partners, and children, including sources of health coverage (e.g., employer-provided, government program, privately funded, etc.).

• Criminal justice module: This module will measure outcomes that are not measured, or not measured well by administrative records including parole, probation violations and recidivism. This section will be administered to all participants in the sites targeting ex-offenders, and only those with a criminal record in the other sites.

• Noncustodial parenting module: These questions will be asked of all participants in the NCP sites and NCPs in the other sites. The focus is on child support payments and parental involvement.

• Social Support and Network module: This module includes questions concerning social support and network composition.

Staff Training

All members of the research teams have extensive experience conducting qualitative and quantitative data collection and analyses. Specific orientation and training on the particular data collection instruments for both studies will be conducted to ensure fidelity to the protocol and consistency of coding.

Purpose of the Data Collection

Findings from this study will be used to inform the Federal government, states, policy makers, program operators and other stakeholders about the efficacy of programs aimed at helping target populations secure and maintain unsubsidized employment.

The exhibit on the following page shows the different timeframes for the 6, 12, and 30-month survey fielding for the sites which have started enrollment.

Revised Figure: 6, 12, and 30-month survey fielding

A.3. Use of Information Technology for Data Collection to Reduce Respondent Burden

Baseline data for the ETJD sites (including the two sites which are also part of STED) are collected via a Management Information System (MIS) developed and managed by ETA. (Collection of baseline data for the ETJD sites received OMB clearance on September 9, 2011 [OMB Control Number 1205-0485].)  Baseline data is collected for the STED-only sites, including the new STED site, via a form which is administered to individual sample members by program staff members in one-on-one or group settings, with program staff members available in all cases to answer any questions. 

Wherever possible, advanced technology will be used in data collection efforts to reduce burden on study participants and on site staff. Staff members in the new STED site, for example, enter potential participants’ responses to the baseline and contact information form into a secure, on-line system.

For the follow-up surveys, the following methods will be used:

1) Computer-assisted survey interviews (CATI/CAPI). This helps to reduce respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length and the need for subsequent call backs. Computer programs enable respondents to avoid inappropriate or non-applicable questions. Key data will be pre-loaded on the surveys based on response to previous survey questions (in the same survey) and earlier surveys (including items collected in the six month and twelve month surveys, where applicable, and at baseline). CATI/CAPI also improves data quality through more uniform administration of the survey questions, more accurate implementation of the skip patterns, and immediate application of range checks, edit checks, and consistency checks of item-by-item responses.¹⁸

2) Social media. Another use of information technology relates to the planned use of social media as a means of communicating with respondents. Social media has had a large impact on how people communicate and provides a flexible means for interaction with respondents. For this reason, we have added questions on the contact information form requesting permission to contact participants or alternative contacts through Facebook. We will use Facebook as means of contact with sample members only if they cannot be contacted via any other means. All contact via Facebook will be done via private messages from the survey contractor and will only be viewable by the sample member. The content of these messages will be a request for the sample member to contact the survey firm via a toll-free number. The private messages will not contain any other information about the sample member, including their status as a member of the STED/ETJD evaluation sample. As with all other personally identifiable information collected as part of the STED/ETJD evaluation, Facebook account information is stored securely with limited access. All project staff with access to such data have received training on the use and handling of personal information and have signed statements affirming their understanding and compliance with the project’s data security procedures.

3) Survey tracking systems. The survey firms have databases that track the location of the participants throughout the project.  Database changes come from mailings to the participants and passively track respondents through the U.S. Postal Service Change of Address database. In addition to being an inexpensive method for being able to contact a respondent, this passive collection approach reduces the need for the respondent to continually provide their most recent address information. It also reduces the probability of seeking alternate contacts in order to find the respondent.

4) Use of Key Survey to administer implementation questionnaires. The research team will work with program sites to determine the most efficient way of administering implementation questionnaires. In some cases, based on preliminary conversations with ETJD program site staff, we expect to administer the questionnaires online using Key Survey. This will minimize burden on the respondent by allowing them to respond to questions at their convenience, without having to schedule time to complete the questionnaires on paper with research staff. Appendix S is a sample screenshot of the program staff questionnaire. The format of the other online questionnaires will be the same as the one provided. We do not plan to administer other data collection instruments online.

5) Integration of other data sources. Finally, when relevant person-level data has been identified as available through an accessible centralized, computerized source, the information has generally been excluded from the proposed data collection package. For example, historical cash assistance (TANF), criminal justice, Food Stamps (SNAP), and UI data will be obtained through administrative records. In addition, all the ETJD sites will use a centralized Management Information System (MIS) maintained by DOL. Information regarding participant eligibility and demographics as well as program participation will be maintained in a single location that is designed for easy access to this information by the research team.

While implementation data collection relies on evaluation staff efforts on-site, we have sought wherever possible to minimize overlap between questions we include in implementation questionnaires and protocols and questions that will be asked through computer-assisted surveys.

A.4. Efforts to Identify Duplication

The information collection will not duplicate information that is already available. Where possible, the evaluations will use available data sources, such as program tracking data, UI wage records, TANF and SNAP data systems, child support systems, and criminal justice records. The surveys will collect data on various other outcomes (such as emotional well-being or detailed job characteristics) that is not available routinely or systematically in program records.

In this particular study, it is quite important that the survey efforts be well coordinated across the two survey firms. MDRC has created a joint data management team on these two projects and has standardized data agreements with both firms. The survey firms will work on separate studies which will ensure that they are not trying to contact the same person. Also, both firms will use identical survey questionnaires. Standardizing the content and format of the data deliverables will streamline work and make it easier to combine the data files at the end of the project.

Finally, a central topic of these evaluations involves the tracking of employment, service participation, and well-being of participants over time. Thus, it is critical to have a core set of consistently worded questions in order to track how the impacts of the program change over time on a common set of measures. However, whenever possible, measures that do not require tracking over time will be assigned to one and only one follow up survey.

A.5 Burden on Small Business

It is anticipated that the data collection activities proposed here will impact small businesses, though all attempts will be made to minimize the burden. Some employers who will be asked to complete the Employer Questionnaire will be small businesses. This brief questionnaire was pre-tested with 9 or fewer people to ensure that the questions are clearly written and understood by respondents. Site visits to employers, which may include small businesses, will be scheduled at the respondents’ convenience, and information about the purpose and scope of the visits will be provided in advance. Interviews will be scheduled with staff who are familiar with the subsidized job program and intend to seek information only about which key respondents have direct knowledge. Data collection activities for the new STED site will not involve small businesses.

A.6 Consequence If Data Collection is Not Conducted

These complementary evaluations represent an important opportunity for the Federal Government to add to the body of knowledge about the impacts of employment-oriented programs for TANF recipients, noncustodial parents, and ex-offenders. This is consistent with the Administration’s strong focus on evidence-based policymaking. If the information is not collected, analyzed, reported, and disseminated, Federal program or policy activities will not be informed by high quality information upon which to base critical decisions regarding future investments.

A.7 Special Data Collection Circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320 (Controlling Paperwork Burdens on the Public). There are no special circumstances that require deviation from these guidelines.

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission

a. Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment on this information collection request through a 60-day Federal Register Notice, published on Friday, March 13, 2015 (FR, Vol. 80, No. 49, pp. 13385-13386). A copy of this notice is attached in Appendix R. No substantive comments were received.

The public was given an opportunity to review and comment on previous information collection requests under this OMB number through the following 60- and 30-day Federal Register Notices: (79 FR 31115), (79 FR 17547) (78 FR 29140), (78 FR 16509), (77 FR 27460), and (77 FR 2550).

b. Consultations Outside of the Agency

All data collection instruments included in this package have gone through extensive review by members of the STED technical work group, ACF and ETA staff, and members of the research team (i.e., MDRC, Abt Associates, DIR, Inc., MEF Associates, and Branch Associates). A list of technical working group members is included in Appendix Q.

A.9 Justification for Respondent Incentives

The respondent gifts of appreciation and justification for each proposed instrument are outlined below.

Previously approved amounts

All ETJD participants will receive a small token of appreciation ($2 each) to encourage them to provide updated contact information in response to the first letter that will be sent to them following random assignment. Sample members will be promised $5 for responding to subsequent requests for updated contact information. Tokens of appreciation for updating contact information are not planned for participants in the five STED-only sites because the ETJD mailings specifically request updated contact information. The STED mailings are: a) a birthday greeting, and b) an advance letter a few days in advance of data collection. The STED mailings mention staying in touch and calling in if they’ve moved, but are not specifically requesting the sample member to fill out a form and send it back in.

6- and 12-month Follow-Up Surveys–Participants who agree to take part in the 6-month survey will receive $25 for completion of the survey. Participants will receive $40 for completion of the 12-month survey. The amounts for the 6- and 12-month survey are based on what was previously approved by OMB in past studies such as the Hard to Employ 15-month follow-up survey, and MDRC’s and the survey firms’ prior experience interviewing similar populations.

30-month Follow-Up Survey - ETJD and STED sample members will receive $50 for completion of the 30-month survey. The amounts for the 30-month survey is based on what was approved by OMB in past studies such as the Hard to Employ 15-month follow-up survey, and MDRC’s and the survey firms’ prior experience interviewing similar populations.

The purpose of the tokens of appreciation is to improve response rates by decreasing the number of refusals, enhancing respondent retention, and providing a gesture of goodwill to acknowledge respondent burdens. This technique is in addition to many of the techniques suggested by OMB to improve response rates that have been incorporated into our data collection effort and are described in Section B3, because our experience has shown that small monetary amounts are useful when fielding data collection instruments with hard-to-employ populations as part of a complex study design. It is essential to include a token of appreciation in order to maximize the response rate, and it is particularly important with a challenging population and a demanding data collection strategy. In a seminal meta-analysis, Singer, et al. (1999) found that incentives in face-to-face and telephone surveys were effective at increasing response rates, with a one dollar increase in incentive resulting in approximately a one-third of a percentage point increase in response rate, on average. They found some evidence that incentives were useful in boosting response rates among underrepresented demographic groups, such as low-income and non-white individuals.¹⁹ This is a significant consideration for this study of subsidized and transitional employment programs. Another important consideration is the burden posed by this data collection, which will take on average 30 minutes of the participant’s time for each 6-month and 30-month surveys and 45 minutes for each 12-month follow-up survey.

The instrument that will be used to collect follow-up data from sample members has unique aspects that make administration difficult and threaten response rates. For this reason we requested and received approval to provide a small monetary amount to all sample members who complete the 6-12-, and 30- month follow-up surveys. Aspects of the data collection effort that also make it more difficult to obtain high completion rates are:

• The surveys include questions that could be perceived as intrusive and therefore could make respondents uncomfortable (i.e., questions about their mental health).

• The subject matter of the interview is not intrinsically interesting to respondents. Moreover, many participants may have negative feelings about the other services received that are of interest, such as welfare, Medicaid, job training, etc.

• Other difficulties in administering the surveys come from the population itself. Educationally and economically disadvantaged groups have been found to be more difficult than the general population to convince to participate in surveys.

The amounts offered increase with each survey wave, for two reasons: 1) the length of the survey increases from the 6-month to the 12-month survey wave; therefore, the survey will likely take the respondent longer to complete than the previous survey; 2) the length of time from the point of random assignment increases with each survey wave; therefore, the survey firm will likely have greater difficulty in tracking down respondents with each survey wave and will need a greater incentive to produce completion rates that will meet or exceed the 80 percent response rate target.

The incentive for the 30-month survey, $50, is $10 more than the amount offered for the 12-month survey. One reason for the increased incentive is the time lapse since random assignment – at 30 months post-random assignment, most sample members may no longer have any connections to the program or operating providers and may no longer feel as connected or obligated to participate in the data collection effort. In addition, given that most respondents will have already completed one or two fairly long interviews prior to being contacted for the 30-month interview, we are concerned that respondent fatigue will complicate efforts to realize adequate response rates. As most of the potential respondents have previously received an incentive for participation, we feel that offering a slightly higher incentive will signal to respondents that we value and appreciate their participation, as well as providing motivation to combat respondent fatigue. In addition, while we have generally been able to achieve high response rates in the earlier survey waves, in some cases we had to extend fielding periods in order reach acceptable response rates. The timeline for the 30-month survey and analysis of the results will limit our ability to extend the fielding period and so timely completion of the interview is of increased importance.

We are aiming to achieve an 80 percent survey completion rate for each follow-up survey. Even with the best data collection practices, it would be very difficult, if not impossible, to obtain such a high completion rate without providing a token of appreciation to participants.

Current Request

No incentives are proposed for data collection in the new STED site.

A.10. Assurances of Privacy

Every effort will be made to maintain the privacy of respondents, to the extent permitted by law. All respondents included in the study will be informed that information they provide will be used only for the purpose of this research. Individuals will not be cited as sources of information in prepared reports. All research staff working on the project have been trained to protect private information and have signed a pledge stating that they will keep all information gathered private to the extent permissible by law. All papers that contain participant names or other identifying information will be kept in locked areas and any computer documents containing identifying information will be protected with a password.

A. 11. Questions of a Sensitive Nature

Questions in all components of the 6-month, 12-month, and 30-month surveys are potentially sensitive for respondents. Respondents are asked about highly personal topics. The questions we have included were selected, in part, because they have been widely used in previous research and are respected among experts. Moreover, since all questions will be pre-tested prior to the survey’s full implementation, if problems arise in regard to any specific items, their inclusion will be reconsidered. Finally, respondents will be informed by program staff prior to the start of the interview that their answers are private, that they may refuse to answer any question, that results will only be reported in the aggregate, and that their responses will not have any effect on any services or benefits they or their family members receive.

The baseline data instrument for the new STED site includes potentially sensitive questions about prior criminal justice involvement, for example, and child support orders. As with the other STED sites, respondents will be informed by program staff that their responses are voluntary and private.

A.12. Estimates of the Hour Burden of Data Collection to Respondents

Previously Approved Information Collections The burden related to previously approved instruments is in Exhibit 1.2. Some adjustments have been made to burden estimates for previously approved instruments, based on actual recruitment experiences to date. Updated burden estimates are included in column 3: Updated Estimate of Total # of Respondents (life of project).

Current request

The current information collection request includes a contact information form and baseline information form for the new STED site. The burden related to the contact information form and baseline information form for the new STED site is in Exhibit 1.3. Total annual burden for all information collection, both approved and requested is in Exhibit 1.4.

Exhibit 1.2: Annual Burden Estimates for Previously Approved Instruments

Instrument	Total # Respondents previously approved	Updated Estimate of Total # of Respondents (life of project)	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hour per Response	Annual Estimated Burden Hours1	Average Hourly Wage of Respondents	Annual Cost
Participant Contact Information Form (5 STED sites)	5,000 (all sample members at five STED-only sites)	7,800	2,600	1	.08	208	$7.25	$1,508.00
Participant Baseline Information Form (5 STED sites)	5,000 (all sample members at five STED-only sites)	7,800	2,600	1	.17	442	$7.25	$3,204.50
Participant STED tracking letters	3,267 (33% of STED sample members, including STED-ETJD shared sites)	No change	1,089	5	.05	272	$7.25	$1,972.00
Participant ETJD tracking letters	1,650 (33% of seven ETJD site sample members)	No change	550	6	.05	165	$7.25	$1,196.25
Participant 6-month survey (Adult sites, subsample)	4,960 (80% of adult STED site sup-sample, including 2 STED-ETJD shared sites)	No change	1,653	1	.5	827	$7.25	$5,995.75
Participant 6-month survey (Young Adult sites)	640 (80% of young adult STED site second cohort)	No change	213	1	.5	107	$7.25	$775.75
Participant 12-month survey (Adult sites)	9,440 (80% of adult STED and ETJD sample members)	No change	3,147	1	.75	2,360	$7.25	$17,110
Participant 12-month survey (Young Adult sites)	2,400 (80% of young adult STED sites)	No change	800	1	.75	600	$7.25	$4,350
Participant Implementation Questionnaire	600 (50 participants /site at all STED and ETJD sites)	No change	200	1	.17	34	$7.25	$246.50
Participant Focus Group Discussion Guide	240 (20 participants/site at all STED and ETJD sites)	No change	80	1	.75	60	$7.25	$435
Program Staff Implementation Questionnaire	120 (10 staff members/site at all STED and ETJD sites)	No change	40	1	.17	7	$19.83	$138.81
Worksite Supervisor Implementation Questionnaire	240 (20 supervisors/site at all STED and ETJD sites)	No change	80	1	.17	14	$23.10	$323.40
Employer Implementation Questionnaire	240 (20 employers/site at all STED and ETJD sites)	No change	80	1	.17	14	$52.21	$730.94
Program Staff Interview Guides	120 (10 staff members/site at all STED and ETJD sites)	No change	40	2	1	80	$19.83	$1,586.40
Program Staff Cost Data Collection Protocol	12 (1 staff member/site at all STED and ETJD sites)	No change	4	1	1	4	$19.83	$79.32
Employer Interview Guides	24 (2 employers/site at all STED and ETJD sites)	No change	8	2	1	16	$52.21	$835.36
Referral Partner Interview Guides	24 (2 referral partners/site at all STED and ETJD sites)	No change	8	2	1	16	$24.64	$394.24
Program Staff Time-Use Worksheet	120 (10 staff members/site at all STED and ETJD sites)	No change	40	1	1	40	$19.83	$793.20
Participant 30-month survey	11,840 (80% of all STED and ETJD participants)	No change	3,947	1	.5	1,974	$7.25	$14,311.50
Estimated Totals						7,240		55,986.92

Exhibit 1.3: Annual Burden Estimates for New Instruments

Instrument	Total Number of Respondents (life of project)	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hour per Response	Annual Estimated Burden Hours1	Average Hourly Wage of Respondents	Annual Cost
Participant contact information form (Paycheck Plus)	4,002	1,334	1	.08	107	$7.25	$776
Participant baseline information form (Paycheck Plus)	4,002	1,334	1	.25	334	$7.25	$2,422
Total					441		$3,198

Exhibit 1.4: Total Annual Burden for OMB #0970-0413

Instrument	Annual Estimated Burden Hours1
All previously approved instruments	7,240
Participant contact and baseline forms	441
Total Annual Burden	7,681

We have estimated the hourly wage of all participants at the federal minimum wage: $7.25 per hour. This is based on our experience with other transitional jobs projects (most of which pay minimum wage) and our expectation that participants will be re-entering the labor market through STED and ETJD programs. Hourly program staff wages ($19.83) were determined using the national mean wage for community and social service specialists from the Bureau of Labor Statistics. We have estimated the hourly wage of worksite supervisors by using the mean hourly wages of first-line supervisors in two industries from the Bureau of Labor Statistics: retail sales supervisors ($19.18) and production industry supervisors ($27.01). Averaging these wages, we have estimated an hourly wage of $23.10 for worksite supervisors. We expect the wages for supervisors in these industries will be comparable to the wages in jobs which will be employing STED and ETJD participants. The hourly wage estimate for employers is based on the mean hourly wage for Human Resource Managers in the Bureau of Labor Statistics: $52.21 per hour. Finally, since many referrals to STED and ETJD programs will be coming from child support, parole, and/or public assistance agencies, we have estimated the hourly wage of referral partners using the national mean wage for Parole Officers from the Bureau of Labor Statistics: $24.64.

Based on the estimated wages, the total annual cost burden to respondents for this information is $55,986.92 as shown in Exhibit 1.2: Annual Burden Estimates. This figure is based on the mean wages for relevant occupations (community and social service specialists, human resource managers, retail and production industry supervisors, and parole officers) reported in the 2010 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates. The addition of the new data collection instruments would bring this total to $59,184.92.

A.13. Other Cost Burden or Respondents and Record Keepers

The proposed data collection will not require the respondents or program staff to purchase equipment or services or to establish new data retrieval mechanisms. Therefore, there are no additional costs to respondents.

A.14. Annualized Cost to the Government.

The estimated annual cost to the federal government for the proposed data collection and analysis is $70,849. The total annual cost for all materials included in this package through the end of the project is $8,039,366. These figures include labor hours, and other direct and indirect costs.

A.15. Reasons for Any Program Changes or Adjustments

This is an additional information collection request under OMB #0970-0413. This amendment includes additional data collection efforts, which include the contact information form and baseline form for the new STED site.

A.16. Tabulation, Analysis, and Publication Plans and Schedule

To determine the effectiveness of the targeted programs, MDRC is collecting four categories of data: 1) baseline data, 2) implementation and process data (some of which will support the cost effectiveness study), 3) surveys of research sample members, and 4) administrative records.

The evaluations will include the following components:

• Implementation and process study: The goal of this component is to understand how the programs operate, who they serve, the context in which they operate, and the dosage of services that sample members receive.

• Impact study: The goal of this component is to assess how the STED and ETJD programs affect employment, earnings, and other key outcomes such as recidivism rates, child support payments, well-being, and family relationships.

• Cost effectiveness study: This component will estimate the net cost per person of the STED and ETJD programs and compare those figures to the financial benefits that the program generate (for example, reduced incarceration or public assistance payments).

Major deliverables for both projects are described below.

ETJD Implementation report (2013). This report served as an introduction to the project, describing the participating sites, the program models, the research sample members, the early implementation experiences, and participation patterns for the program group.

ETJD Interim impact report (2015). This report will summarize the implementation data from the early report, but will focus mostly on one-year impacts, drawing from both administrative records and the 12-month survey. At this early follow-up point, the report will focus heavily on differences between groups in service receipt and the initial transition from subsidized to unsubsidized employment.

ETJD Final impact report (2017). This report will present results of the data collection for all participants from the second follow-up survey and administrative records as well as the cost-effectiveness analysis. The report will communicate the evaluation findings to a broad audience.

ETJD Public Use File documentation. This report will serve as documentation for the final data sets used by the evaluation, including instructions for data retrieval, code books presenting means and frequencies for all variables, and important decisions made in construction of outcome variables.

STED Profile Report (2015). This report will provide detailed profiles of sites selected to participate in the STED evaluation.

STED Baseline Report (2015). This report will describe the project sites, the random assignment process, and characteristics of the sample.

STED 12-Month Report (2017). This report will describe findings from the implementation study and report impacts on all key outcomes from administrative records and the 12-month survey. We propose to produce a relatively brief main report, with technical appendices providing more detailed tables and explanations

STED 30-Month Report (2019). This report will describe findings from the implementation study and report impacts on all key outcomes from administrative records and the 30-month survey as well as provide a summary of the benefit-cost findings.

STED Paycheck Plus interim report (2019). This report will document take-up of the expanded credit for the first two years and report early impacts on employment, earnings, and child support payments.

STED Paycheck Plus final report (2020). This will document take-up of the credit for all three program years and impacts on employment, earnings, benefit receipt, and child support payments.

A.17. Reasons for Not Displaying OMB Approval Expiration Date

The expiration date for OMB approval will be displayed on all forms completed as part of the data collection.

A.18. Exceptions to Certification Statement

No exceptions are necessary for this information collection.

1 Revised to include alternate survey instruments which were developed for the youth/young adult STED sites.

2 Revised to include 30-month Follow-up survey and to amend burden estimates to reflect actual study enrollment.

3 Revised to include the baseline form and the participant contact information form to be used for the new STED site. Edits and updates related to this revision are highlighted yellow.

4 Autor et al., 2008.

5 www.stateofworkingamerica.org, Table 4.14.

6 Ibid, Table 4.15.

7 http://www.bls.gov/cps/cpsrace2010.pdf

8 http://nces.ed.gov/programs/digest/d12/tables/dt12_432.asp

9 See, for example, Rep. Rangel’s proposal (HR 3970), Senate Bill 1626, introduced by Senators Bayh and Obama in 2007, Berlin, 2007, Edelman et al., 2006 and Scholz, 2007.

10 Eissa and Hoynes, 2006; Holt, 2006; Dahl and Lochner, 2008.

11 Berlin (2000).

12 This and other transitional jobs projects are discussed in more detail in Appendix A.

13 The first survey will be administered while research group members are engaged in the study program or soon thereafter and so the exact timing of administration will vary from three to nine months depending on the length of the intervention in each site.

14 One of the STED sites, the Chicago Bridges and Pathway Initiative, is a formative evaluation and will have a smaller sample size and less intensive data collection effort. Specifically, one follow-up survey will be administered to site participants at 6-9 months after random assignment.

15 The tracking letter that is included in Appendix E is an example of the first letter that will be sent to participants. Subsequent letters will be nearly identical in content, but will be updated to reflect the number of months since random assignment and will promise $5 for responding.

16Unlike the Participant Questionaire discussed earlier, the 6-month survey will be administered to both the program and control groups. Thus, it focuses on measuring differences between the groups.

17 Employment history since random assignment was also collected in the 12-month survey. Due to differential interview timing (i.e., not all 12-month respondents were or will be interviewed at precisely 12 months after random assignment) and the possibility that a respondent to the 30-month survey did not complete a 12-month follow-up interview, employment history since random assignment will be re-collected on the 30-month survey.

18 Another benefit of CATI/CAPI is that interviewers can focus on the respondent rather than management of the survey instrument, creating a more pleasant experience for the respondent. The technology ensures that scheduled appointments are honored with respondents. CATI/CAPI also ensures adherence to dialing protocols, maintaining the integrity of the study without unduly burdening sample members and eliminates many human errors, such as accidental calling of resolved sample records. Finally, CATI provides translated scripts for crisp script delivery to non-English speakers.

19 Berlin, M., L. Mohadjer and J. Waksberg (1992). An experiment in monetary incentives. Proceedings of the Survey Research Section of the American Statistical Association, 393-398; de Heer, W. and E. de Leeuw. “Trends in household survey non-response: A longitudinal and international comparison.” In Survey Non-response, edited by R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little. New York: John Wiley, 2002, pp.41-54; Singer, E. and Kulka, R. Studies of Welfare Populations: Data Collection and Research Issues, Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs. Ploeg, Robert A.Moffitt, and Constance F.Citro, Editors. National Academies Press, Washington, DC, 2000, pp. 105-128.