REO_OMB Package Part B_rev_Apr2019_clean

part B: Reentry Employment Opportunities (REO) Evaluation

OMB No. 1290-0NEW

april 2019

^{OFFICE OF MANAGEMENT AND BUDGET SUPPORTING STATEMENT PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS for the} Reentry Employment Opportunities (REO) evaluation

The U.S. Department of Labor’s (DOL) Chief Evaluation Office (CEO) is undertaking the Reentry Employment Opportunities (REO) Evaluation. The overall aim of the evaluation is to determine whether the REO programs improve employment outcomes and workforce readiness for young adults and adults with previous criminal justice system contact. CEO contracted with Mathematica Policy Research and its subcontractor, Social Policy Research Associates (SPR), to conduct this evaluation. The evaluation will include an impact study and an implementation study. This package requests clearance for two data collection instruments (instruments included as supporting documents):

1. Grantee survey

2. Baseline information form (BIF)

These data collection instruments have been reviewed and approved by Health Media Lab Institutional Review Board.

B.1. Respondent universe and sampling methods

The universe of potential grantees for the data collection efforts for which OMB clearance approval is requested includes the Training to Work (T2W) grantees that received grants in 2015 and 2016, the Reentry Demonstration Project (RDP) grantees that received grants in 2016, the Reentry Project (RP) grantees that received grants in 2017, and the RP grantees that received grants in 2018 (Table B.1).¹ The implementation study will include up to 119 grantees and subgrantees from all three types of grants.² The impact study will include up to 10 of the REO RP grantees and subgrantees; after the impact study sites are identified, a purposively selected subset of them (assumed for now to be 5 out of a maximum of 10) will be part of a follow-up survey data collection effort with impact study participants. This section outlines the respondent universe, proposed methods for respondent selection, and the expected response rates for each of the two data collection activities being cleared as part of this submission.

Table B.1. Respondent universe by grant year and type

Year	Grant Stream	Grantees	Sites
2015	Training to Work 3	22	22
2016	RDP	7	26
2016	Training to Work 4	17	17
2017	RP CBO	23	23
2017	RP Intermediary	9	42
2018	RP CBO	32	32
2018	RP Intermediary	9	39
	Totals	119	201

CBO = community-based organization. RDP = Reentry Demonstration Project. RP = Reentry Project.

Implementation study grantee survey. The goal of the implementation study grantee survey is to gather common information about organizational settings and intervention characteristics for REO grantees. The survey will be distributed to an executive director from the census of 97 grantees that received REO grants in 2016, 2017, or 2018. No statistical methods will be used in selecting grantee survey respondents because all grantees that received grant awards during years 2016, 2017, and 2018 will be included in the survey fielding effort. The study team anticipates that the survey will take an average of 20 minutes to complete. Therefore, a total of 97 grantees will be included in the survey response effort. The study team plans to administer the survey to the universe of program managers and anticipates a 100 percent response rate for the survey. Participation in the evaluation is a condition of grant award from DOL allowing for a 100 percent response rate.

Impact study BIF. Study design considerations, as well as resources for data collection and analysis, will influence the number of grantees included in the impact study. DOL and the study team will determine which grantees will be included. Given the study’s resources, it is expected that no more than 3,780 sample members will be enrolled in the impact study. If the most suitable grantees for inclusion in the impact study are expected to serve a relatively large number of participants, then there will be fewer grantees in the impact study than if the most suitable grantees for inclusion in the impact study serve a relatively small number of participants. Thus, although up to 10 grantees might be included in the impact study, the number of grantees might be smaller.

In sites selected for the impact study, all applicants for program services (after OMB approval is received and study procedures commence at the site) will complete a BIF before they are randomly assigned to either a program group that will be able to receive REO services or a control group that will not. The universe of REO study participants (which will include program and control group members) across up to 10 impact study sites is estimated to be 3,780. The study team anticipates collecting information on study participants’ characteristics for 100 percent of the REO study participants. The BIF form will be administered before enrollment into the study, allowing for a 100 percent response rate.

Table B.2 summarizes this information, including the sampling method, estimated sample sizes, and assumptions about response rates for each data collection activity and respondent type by site.

Table B.2. Sampling and response rate assumptions, by data collection activity and respondent type

Respondent type	Sampling method	Universe of potential respondents	Estimated selected sample	Average responses (per site)			Estimated responses per respondent		Estimated responses (across sites)
Grantee Survey
Grantee survey respondentsa	Census	97	97		1	1		97
Impact Study Baseline Information Form
Study participantsb	Census	3,780	3,780		378	1		3,780

^a The grantee survey will be administered to up to 97 grantees because there were 97 grants awarded during years 2016 through 2018 (24 in year 2016, 32 in year 2017, and 41 in 2018).

^b The BIF will be completed by a census of impact study participants. It is possible that sites will begin enrolling program participants before random assignment for the impact study begins. Therefore, it is possible that the census of program participants will be larger than the census of impact study participants who will complete the BIF. Study resources suggest that a maximum of 3,780 study participants may be included in the impact study. Study design considerations, as well as resources, will influence the number of sites included in the impact study. Given that the maximum number of grantees that could be included is 10, the average number of respondents per site is shown to be 378. However, if fewer grantees are included in the study, then the number of study participants per site might be larger. Based on previous experience administering BIFs, the study team anticipates that 100 percent of people who participate in the study will complete a BIF.

B.2. Procedures for the collection of information

Understanding the effectiveness of the REO grants requires data collection from multiple sources. For the implementation study, data collection will include telephone interviews, individual-level program data, a grantee survey, and virtual focus groups of grantees. At the sites included in the impact study, the study team will collect baseline information on study participants at program application to understand the baseline characteristics of the sample. Outcome data will come from a follow-up survey of a sample of study participants, as well as administrative earnings data and criminal justice system data for all impact study participants. The data collection activities covered by this clearance include the grantee survey and the BIF. The next section describes each of these data collection activities.

Grantee survey. As part of the implementation study, an electronic survey will be fielded to up to 97 grantees to obtain information on the REO grantees’ approaches to project management, recruitment and outreach, and service delivery. This survey will include a set of common questions to lead to insights about variations across grantees and grant programs and to put the impact and implementation study data in context.

Baseline information form (BIF). As part of the impact study, at study intake, individuals who request being served by the grantees and subgrantees included in the impact study will provide demographic characteristics and other information on BIFs. In addition to sample member contact information, the BIF will also collect information on the baseline characteristics of sample members, such as sex, race/ethnicity, education level, and prior justice system involvement. In addition, the BIF will capture social network use and alternative contact information for up to three relatives or friends who might know how to contact the sample member. Whenever possible, BIFs will be collected electronically through the study’s web-based random assignment system (RAS). The baseline information, but not the contact information, will be analyzed using either t-tests (such as for binary variables) or chi-squared tests (such as for categorical variables) to ensure that the program and control groups are not systematically different from each other on these characteristics, which are measured prior to random assignment.

Consent to participate in the research study will be obtained from all impact study participants. To fully ensure informed consent, the study team will collect written consent from all REO evaluation participants. Written consent forms will describe the purpose of the study; outline the information that will be collected; explain the risks, benefits, and voluntary nature of participation; and collect participants’ consent to be included in the evaluation. The participant consent form for the impact study is included in the Supplementary Documents.

1. Statistical methodology, estimation, and degree of accuracy

Implementation study grantee survey. The main type of data collected from the grantee survey will be contextual information about the program, services offered, and any unique approaches used or populations served. No complex statistical methodology (such as sample stratification) or estimation will be needed in the analysis of the grantee survey data. The data will be analyzed using simple descriptive measures to generate aggregated counts of responses. Responses to open-ended questions will be coded to identify key themes across sites and enable the study team to describe the REO grantees’ characteristics and experiences.

BIF. The data gathered through the BIF will be tabulated using descriptive methods (including simple frequencies, cross-tabulations, and means, when appropriate) to provide contextual information about the characteristics of participants in the impact study. If a random assignment design is conducted properly, there should be no systematic observable or unobservable differences between research groups across key indicators including sex, race/ethnicity, education level, and prior justice system involvement, except for the services offered after random assignment. Therefore, the BIF data will be used to assess whether randomization was conducted properly by conducting statistical tests, such as t-tests or chi-squared tests, to assess whether there are statistically significant differences in these baseline measures of program and control groups using available baseline data. Because BIF data will be collected before random assignment, there should be no differences in data quality or response across the program and control groups. The study team will conduct t-tests or other tests on each baseline measure in isolation and will also conduct a joint F-test to assess the joint significance of all the differences in baseline characteristics.

2. Unusual problems requiring specialized sampling procedures

Specialized sampling procedures are not required for administration of the grantee survey or BIF. The study team will attempt to collect grantee survey data from all grantees awarded grants during years 2016 through 2018 and BIF data from every impact study sample member who is from the sites that will be part of the follow-up survey data collection efforts.

3. Periodic data collection cycles to reduce burden

The grantee survey for the implementation study will be administered only once for any individual respondent. The BIF will be administered only once for any individual grantee respondent.

B.3. Methods to maximize response rates and minimize nonresponse

As their grant agreements indicate, REO grantees and subgrantees included in the evaluation are expected to participate in the data collection activities as a condition of their grant awards.

Grantee survey. The study team expects to achieve a response rate of 100 percent for the grantee survey. As described above, an agreement to participate in the evaluation is a condition of receipt of REO grant funds. In addition, the study team will work with DOL to obtain its help in promoting full engagement of the grantees. The survey will be designed to be as brief as possible, with clear, easy-to-answer questions (mostly closed questions, with a few open-ended questions). An official advance letter with log-in information will be mailed to grantee sample members to legitimize the study and encourage participation. Finally, the data collection plan will include a two-stage outreach strategy for reluctant responders. Beginning in week 2 of data collection, an email reminder will be sent to those who have not responded. Beginning in week 4, the study team staff will conduct follow-up telephone calls (in addition to email reminders) to nonresponders.

BIF. The baseline information forms will be collected from all eligible individuals at the sites selected for the impact study and entered into a web-based system as they go through an intake process. The methods to maximize response for the intake forms will be based on approaches used successfully in many other random assignment studies (for example, the YouthBuild Evaluation, for which nearly 4,000 people were randomly assigned) to ensure that the study is clearly explained to study participants and staff and that the forms are easy to understand and complete. Staff will be thoroughly trained on how to address study participants’ questions about the forms. Grantee staff will also receive a site-specific operational procedures training prepared by the research team, contact information for members of the research team, and detailed information about the study.

Methods to ensure data reliability. Several well-proven strategies will be used to ensure the reliability of data collected from the grantee surveys, key informant interviews, and BIFs.

• Grantee survey. The study team has used a systematic approach to developing the instrument to ensure the quality of the data. All respondents will be assured that reports will never identify respondents by name.

• BIF. The same BIF will be used across all impact study sites to ensure consistency in the collected data. The evaluation team will have reviewed the forms extensively, and the forms were thoroughly tested in a pre-test involving nine individuals from nonparticipating sites. Staff will be trained on each item on the BIF to ensure that they understand each item and will accurately record the information provided by program applicants. In addition, each participating site will have access to the web-based system for entering the information from the BIF. To ensure complete and accurate data capture, this platform will flag missing data or data outside a valid range. Staff will be trained to probe for valid responses if a respondent leaves an item missing. Furthermore, to ensure that the most critical pieces of information are collected on everyone, the study team will program the web-based system to not allow missing answers for critical items. At the analysis stage, the study team will create binary variable flags for all noncritical items with missing values and include these flags in the analyses. In addition, staff will be trained on project security, including safeguards used to protect personally identifying information while collecting and storing sample member information.

B.4. Tests of procedures or methods to be undertaken

Grantee Survey. During formal pretesting, the study team tested the grantee survey with nine respondents from nonparticipating sites. The study team sent respondents the survey via email and asked them to complete it, scan it, and return it to Mathematica by email. Respondents were instructed to keep a copy of their completed survey to reference during the respondent debrief. The debriefs were conducted by telephone using a semi-structured interview protocol. Feedback from the pretests was used to clarify the wording of the instructions and questions to eliminate questions that respondents found overly burdensome. The updated grantee survey is included as a supporting document.

BIF. The BIF was tested with nine individuals from the year 2017 grantees so that the pretest participants most closely approximated the future study sample. The pretest sample included respondents from an adult, Training to Work program, and a young adult, Reentry Project and the sample was diverse in terms of age, work, and incarceration experiences. The BIF pretests were conducted in-person to ensure that the study team could gauge participants’ reactions and ease of completing and understanding the instrument. For example, the pretests allowed the study team to observe whether the respondents took longer to answer specific questions or if they gave other non-verbal signs that they found specific questions difficult to understand or answer. Each pretest respondent was asked to complete the BIF, and afterwards, a member of the study team debriefed with them to get their overall comments and to further assess their understanding of key questions and terms. Feedback from the BIF pretests was used to clarify the question text and remove questions that respondents consistently left blank. The updated BIF is included as a supporting document.

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data

Consultations on the statistical methods used in this study were part of the study design phase to ensure technical soundness. The individuals listed in Table B.3 were consulted on the statistical methods discussed in this submission to the Office of Management and Budget.

Table B.3. Individuals who were consulted for the REO evaluation

Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393	Project director	Dr. Karen Needels Senior Researcher (609) 750-4043
(609) 799-3535	Survey director	Dr. Jillian Stein Survey Researcher (609) 716-4395
	Site selection task lead	Dr. Johanna Lacoe Researcher (510) 285-4618
	Quality assurance reviewer	Dr. Jillian Berk Associate Director (609) 275-2243
Social Policy Research Associates 1330 Broadway, Suite 1426 Oakland, CA 94612	Principal investigator	Dr. Andrew Wiegand President, CEO, and Principal (510) 763-1499 (x 636)
(510) 763-1499	Implementation study task leader	Ms. Hannah Betesh Senior Associate (510) 788-2469
	Administrative data collection task lead	Mr. Christian Geckeler Senior Associate (510) 788-2461

Table B.4 lists staff responsible for overseeing the collection and analysis of data.

Table B.4. Individuals who will oversee the collection and analysis of data for the REO evaluation

Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393	Project director	Dr. Karen Needels Senior Researcher (609) 750-4043
(609) 799-3535	Survey director	Dr. Jillian Stein Survey Researcher (609) 716-4395
	Site selection task lead	Dr. Johanna Lacoe Researcher (510) 285-4618
	Quality assurance reviewer	Dr. Jillian Berk Associate Director (609) 275-2243
Social Policy Research Associates 1330 Broadway, Suite 1426 Oakland, CA 94612	Principal investigator	Dr. Andrew Wiegand President, CEO, and Principal (510) 763-1499 (x 636)
(510) 763-1499	Implementation study task leader	Ms. Hannah Betesh Senior Associate (510) 788-2469
	Administrative data collection task lead	Mr. Christian Geckeler Senior Associate (510) 788-2461

1 This clearance request does not include year 2015 grantees because it is expected that their grants will expire prior to receipt of approval for the request. Hence, this package requests clearance for implementation study data collection for 97 of the 119 grantees included in the overall study.

2 A subset of the REO grants were awarded to intermediary organizations that competitively selected subgrantees. Parts of the evaluation will include direct grantees and the subgrantees of the intermediaries.

B.4