Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Next Generation of Enhanced Employment Strategies Project
OMB Information Collection Request
0970 - 0545
Supporting Statement
Part A
AUGUST 2024
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Marie Lawrence
Sarita Barton
Megan Reid
Part A
Executive Summary
Type of Request: This Information Collection Request is for a nonsubstantive change to the information collection originally approved in April 2020 under OMB #0970-0545 with an extension approved in April 2023.
Progress to Date: Information collection for the Next Generation of Enhanced Employment Strategies (NextGen) Project was originally approved in April 2020.
Currently, the NextGen Project is actively collecting data from four impact study programs, including ongoing collection for the first and second follow-up surveys for NextGen study participants.
Timeline: The two follow-up surveys are currently live. Once this change is approved, the study team will modify the surveys immediately.
Previous Terms of Clearance: There were no terms of clearance for the previous approvals of this data collection.
Summary of changes requested: The study team seeks approval to collect NextGen study participants’ Social Security numbers on the first and second follow-up surveys. As approved, the study team already attempts to collect Social Security numbers from study participants during the enrollment process using the approved identifying and contact information data collection (Instrument 2). However, some participants do not provide their Social Security number during enrollment. The study team would like to collect their Social Security number during the first and second follow-up surveys (Instruments 3 and 4). Collecting this information will ensure that the project team can conduct analysis on administrative records for a larger percentage of the study sample.
Time Sensitivity: Since data collection is in process, we request a response as soon as possible.
A1. Necessity for Collection
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) seeks approval to continue data collection activities for programs and participants in the Next Generation of Enhanced Employment Strategies (NextGen) Project. OPRE has spent decades studying strategies to help low-income people find and keep jobs. Findings from these studies have been mixed, revealing variation in what works for whom and the duration and magnitude of impacts. Some studies have also demonstrated that certain programs are less accessible to individuals with complex challenges, such as low educational attainment or involvement with the criminal justice system, due to the program’s eligibility requirements.
The NextGen Project is intended to build on the findings and lessons learned from these past and ongoing evaluations by identifying and rigorously evaluating the “next generation” of employment strategies for highly vulnerable populations with complex barriers to obtaining and retaining employment. These strategies may be enhancements or adaptations of previously evaluated strategies, or innovative approaches showing promise in the field and ready to be tested. Additionally, the project has a particular interest in the role of market-oriented, employment-focused programs in assisting highly vulnerable populations obtain and retain employment. The current data collection request is necessary to continue and complete these rigorous evaluations.
A2. Purpose
Purpose and Use
The information collected through this information request will be used to evaluate innovative programs serving low-income individuals facing complex challenges to employment and economic independence to expand the evidence base in this area. The NextGen Project is coordinating with another current project sponsored by OPRE, the Building Evidence on Employment Strategies for Low-Income Families (BEES) study (OMB #0970-0537). BEES includes studies of 20 employment-focused programs; these do not overlap with the programs selected for the NextGen Project. The NextGen Project and BEES have a common goal to foster stronger understanding of the types of programs that can improve labor market outcomes for low-income individuals; however, the projects also maintain separate domains of focus. In addition, both projects are involved in a joint effort with the Social Security Administration (SSA). SSA has provided demonstration program funds to ACF to support the addition of a disability focus in both projects; specifically, to identify and evaluate employment-related programs for potential SSI applicants. This is intended to assist SSA in better understanding the types of early interventions that effectively connect or reconnect potential SSI applicants to work before they apply for SSI. See Section A4 for information about coordination and efforts to not duplicate activities.
Data collection instruments for the NextGen Project impact studies provide baseline and outcome data about study participants, which the project team will use to estimate the effectiveness of each program. The project team will use data collection instruments for the descriptive studies to describe each program’s design, staffing, service provision, partnerships, and other details necessary to understand the nature of and context for the programs, and for other organizations to replicate them. The instruments will also help inform the interpretation of impact findings. Finally, the project team will use data collection for the cost studies to estimate the costs of implementing each evaluated program and to estimate the cost-effectiveness of the programs. The results will provide policymakers and practitioners with high-quality information on the effects, design and implementation, and the cost of the programs. Having this information will help strengthen policy and practice to better serve individuals facing complex challenges to employment and economic independence. Study findings may also inform future studies in this area. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.
Research Questions or Tests
The questions this evaluation will answer are in Table A.1.
Table A.1. Research questions for the NextGen Project
Impact studies |
Did the program affect the amounts and types of services participants receive? |
Did the program improve participants’ employment outcomes (employment, earnings, job retention and advancement, and quality of job) and economic independence (income, public assistance receipt)? |
Did the program improve outcomes relevant to the challenges faced by the target population, for example reduce substance abuse; reduce criminal justice involvement; or increase education, credentialing, and training? |
Did the program improve participants’ physical health, mental health, and well-being? |
Was the program more effective for some groups of participants than others? If so, which groups? |
Did the impacts of the program change over time? If so, how? |
How did the program’s costs compare to the benefit of the impacts it generated? What were the net benefits for participants and society as a whole? |
Descriptive studies |
How was the program designed and implemented? |
What contextual, organizational, and other factors impeded or facilitated implementation? |
What were the challenges faced, solutions, and lessons learned? |
What were the characteristics of study participants? |
What services were participants offered, and what were the participation and outcome patterns? |
What role did employers play in the program? How do local labor market conditions affect the program design, implementation, and employers’ and participants’ involvement? |
Which program services or implementation features appear to be related to program impacts? Which components or services do participants and staff perceive to be helpful? |
What were the backgrounds and experience of program staff and program leaders? |
How did staff spend their time, and how many participants did they work with? |
How did program leaders spend their time? |
How did participants perceive the program? What were the most helpful elements? How did the program affect their lives? |
Cost studies |
How was the program funded? What were its costs? Was the program sustainable? |
Study Design
The NextGen Project includes experimental impact, descriptive, and cost studies. It is studying programs that include a wide range of supports designed to serve individuals with multiple challenges to employment and that might be delivered by public–private partnerships, interagency collaborations, government initiatives, or nonprofit agencies.
The impact studies are intended to produce internally valid estimates of the program’s causal impact, not to promote statistical generalization to other sites or service populations. The descriptive and cost studies are intended to present internally valid descriptions of the service population, implementation, and cost of the programs in the chosen sites, not to promote statistical generalization to other sites or service populations. See Section B.1 of this information collection request (ICR) for further information about the appropriateness of the design and its limitations.
Four programs have been identified for inclusion in impact studies for the NextGen Project. The programs were assessed to determine if they meet three general criteria: (1) the program addresses the research priorities of this project; (2) the program is well implemented, or could be after some technical assistance; and (3) a rigorous evaluation of the program is feasible, using an experimental design, or could be after the program receives some technical assistance. Additionally, included programs have some evidence that they might be effective, and so an evaluation of the program builds on existing evidence and is valuable to the field. Programs were also selected to address SSA’s research interests. In addition to the programs participating in impact studies, three additional programs are participating in descriptive and/or cost studies. The programs studied are not national programs, and the study is not designed to be nationally representative, nor will the project team attempt to generalize the evaluation results beyond the programs and target populations under study. The NextGen Project is not actively recruiting additional programs.
Phased Approach to Data Collection Approval
The NextGen Project used a two-phased approach for previous OMB approvals. This began with an initial request that included Phase 1 instruments for approval and draft Phase 2 instruments along with estimated burden that we anticipated would need to be tailored based on final recruitment of programs. We did not seek approval at that time for Phase 2 instruments; instead, we indicated that, under Phase 2, we would request approval of the remaining instruments. Phase 2 instruments were also included in the Federal Register Notices, allowing for public comment on the initial versions.
Phase 1
In Phase 1, the project team recruited the programs based on information that had been collected to inform this study through the umbrella generic, Formative Data Collections for ACF Research (OMB #: 0970-0356). Following OMB approval in April 2020, the project team began to administer the baseline survey (Instrument 1) and to collect identifying and contact information for study participants (Instrument 2). These baseline data collections are uniform across programs selected for evaluation except for the program-based skip logic in the instruments. OMB later approved of revisions to these instruments.
Instrument 1. Baseline survey – revised
Instrument 2. Identifying and contact information – revised
Phase 2
Once programs were selected for the evaluation, we submitted updated materials and burden estimates as either non-substantive change requests or revisions with abbreviated public comment time, dependent on the level of changes and guidance provided by the OMB Office of Information and Regulatory Affairs.
Instrument 3. First follow-up survey – revised
Instrument 4. Second follow-up survey – revised
Instrument 5. Service receipt tracking – revised
Instrument 6. Staff characteristics survey – revised
Instrument 7. Program leadership survey – revised
Instrument 8. Semi-structured program discussion guide – revised
Instrument 9. Semi-structured employer discussion guide – revised
Instrument 10. In-depth participant interview guide – revised
Instrument 11. Cost workbook
Impact studies. The experimental impact studies will provide rigorous evidence on whether each program is effective, for whom, and under what circumstances. Participants eligible for the programs are asked to consent to participate in the study (Appendix A) and, if they provide consent, are randomly assigned to one of two groups: a treatment group offered the program or a control group not offered the program. Members of all study groups continue to have access to other services offered in the community. Individuals who do not consent to participate in the study are not randomly assigned, do not participate in the data collection efforts, and are not eligible to receive the intervention (until after the second follow-up survey has been fielded).
The project team is collecting information from study participants for the impact studies at three points: (1) at program entry before random assignment occurs (baseline); (2) at about 6 to 12 months after random assignment via the first follow-up survey; and (3) at about 18 to 21 months after random assignment via a second follow-up survey. The timing of the follow-up surveys varies depending on when each program’s theory of change suggests impacts might be expected. Baseline data collection and first follow-up survey administration are currently underway. The second follow-up surveys began in April 2023.
Descriptive studies. The descriptive study for each program will describe the following: (1) the community, economic, and program context in which the program operates; (2) the characteristics of the program model, including the target population, services offered, role of partners and employers, theory of change, and plans for sustainability and replication; and (3) the implementation and cost drivers of the program, such as leadership, organizational culture and structure, staffing and staff development, and service delivery. The data collection period for the descriptive study varies by participating program but is typically beginning around 6 to 12 months after the study begins enrolling participants. If respondents consent to being recorded, the interviewer audio records discussions with program administrators, supervisors, staff; key partner staff, including employers; and participants. Data collection for the descriptive studies is currently in process.
Cost studies. The cost study for each program (1) provides descriptive information about the amount, sources, and types of its funding, and (2) will produce an estimate of the average cost of the program per participant. The average cost of the program per participant will be used in the benefit-cost analysis. In that analysis, the benefits that accrue to program participants such as increased earnings and reduced receipt of public benefits will be compared with the cost of providing program services. The study team is conducting data collection for the cost studies around the same time as the data collection for the descriptive studies. Data collection for the cost studies is currently in process.
The impact, descriptive, and cost study data collections are included in Table A.2.
Table A.2. Data collection activities for the NextGen Project
Data Collection Activity |
Instruments |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Baseline data collection (impact study)
|
Instrument 1: Baseline survey – revised
Instrument 2: Identifying and contact information – revised |
Respondents: All consenting study participants.
Content: Baseline survey includes information on demographics, receipt of Social Security Administration benefits, employment history, social trust, COVID-19-related challenges, and challenges to maintaining employment. Identifying information includes name, Social Security number, and date of birth. Contact information includes physical and electronic addresses and social media information for participants and up to three friends or relatives. Instrument 2 also includes the Center for Epidemiologic Studies Depression Scale Revised (CESD-R).
Purpose: Baseline survey data will be used to describe the study sample and check that the characteristics of the study participants are similar on average across groups. The data will also be used to define subgroups, as covariates in regression models, and for weighting for nonresponse. A question-by-question justification for the items included in the baseline survey is presented in Appendix B – revised. Identifying information are used before random assignment to make sure participants have not already been enrolled in the study. The project team will use this information later to match study participants to their administrative data records to assess outcomes. In addition, the team will collect detailed contact information to locate participants to complete follow-up surveys.
|
Mode: Baseline survey allows for multiple administration options: by program staff, self-administered by study participants via the web, or by NextGen Project staff via telephone. RAPTER® identifying and contact information and responses to CESD-R questions are provided verbally by study participants and entered into RAPTER® by program staff.
Duration: 25 minutes (total to complete the baseline survey and provide identifying and contact information) |
Follow-up data collection (impact study)
|
Instrument 3: First follow-up survey – revised August 2024
Instrument 4: Second follow-up survey – revised August 2024 |
Respondents: The project team will attempt to survey all study participants.
Content: The follow-up surveys collect data on outcomes of interest, including service receipt, employment, earnings, economic independence, well-being, health status, substance use, involvement in the criminal justice system; perceptions of the usefulness of the program being evaluated (for treatment group only); and updated contact information (on first follow-up survey only). They collect Social Security numbers if the study team did not collect one during during study enrollment. The exact questions asked vary by site depending on the site’s target population.
Purpose: The project team will use survey data to estimate program impacts on outcomes of interest; estimate the program impacts on the services the study participants receive; describe treatment group members’ perceptions of the usefulness of the program being evaluated; and describe the study sample. The updated contact information from the first follow-up survey will be used to assist in locating study participants for the second follow-up survey. A question-by-question justification for the items included in the follow-up surveys is in Appendix D.
|
Mode: Participants self-administer via the web. Alternatively, administered by NextGen Project staff via telephone.
Duration: 50 minutes per follow-up survey |
Treatment group service receipt (descriptive study)
|
Instrument 5: Service receipt tracking – revised |
Respondents: Program staff
Content: Information about the treatment group members’ participation in the program. In programs that also provide services to control group members, program staff might also record information on receipt of services of control group members.
Purpose: To describe the service receipt of treatment group members, including type of service, duration, location, and mode. |
Mode: Program staff enter information about services received by study participants through the program in RAPTER®. If a program already collects data on service receipt through its own database, the study uses the information the program already collects.
Duration: 5 minutes per entry |
Characteristics of program staff and leaders (descriptive study)
|
Instrument 6. Staff characteristics survey – revised
Instrument 7. Program leadership survey – revised |
Respondents: Program staff and leaders.
Content: Staff members’ and leaders’ professional backgrounds, skills, experience, credentials, and perceptions of the program. Leaders’ resource investments and decision-making processes. Changes due to COVID-19.
Purpose: To provide insight into how program structure, staffing, and leadership might affect implementation of the program. Compared with the semi-structured interviews, described below, the surveys will enable the collection of information (1) in a more structured format, (2) on topics that staff and leaders might be uncomfortable talking about in a group setting, and (3) from a broader set of staff and leaders than would have the time to participate in a semi-structured interview. |
Mode: Program staff and leaders self-administer the surveys via the web.
Duration: 25 minutes for staff survey; 15 minutes for leadership survey |
Discussions with program staff, partners, and employers (descriptive study)
|
Instrument 8. Semi-structured program discussion guide – revised
Instrument 9. Semi-structured employer discussion guide
|
Respondents: Program administrators, supervisors, staff; key partner staff, including employers
Content: Semi-structured discussions with program administrators, supervisors, direct service staff, community partners, and specialized treatment providers will provide information about the program’s design and implementation and any COVID-19 related challenges. Semi-structured discussions with employers will collect information about their involvement in developing and executing the programs of interest.
Purpose: To describe each program’s design, staffing, service provision, partnerships, and other details necessary to understand the nature of and context for the programs, and for other programs to replicate them. Also to help inform the interpretation of impact findings.
|
Mode: The interviews are conducted in person during site visits, either individually or in small groups. Interviews may also be conducted via telephone or video dependent upon any COVID-related restrictions.
Duration: 90 minutes per administrator; 60 minutes per program supervisor, key partner staff, employer, or direct service staff. |
In-depth participant interviews (descriptive study)
|
Instrument 10. In-depth participant interview guide – revised
|
Respondents: Select study participants
Content: Participants’ background and goals, experiences and challenges finding and retaining employment, experiences with the program, including reasons for disengaging from the program, if applicable. Challenges related to COVID-19.
Purpose: To provide the “stories” that will make the findings from the implementation and impact studies more meaningful. They might also inform the understanding of whether the program was implemented as planned and suggest possible refinements. |
Mode: The interviews are conducted in person during site visits. Interviews may also be conducted via telephone or video dependent upon any COVID-related restrictions.
Duration: 120 minutes |
Cost data collection (cost study)
|
Instrument 11. Cost workbook |
Respondents: Program leader (or a designee)
Content: Excel-based cost workbook to record information on the expenditures associated with the program for a recent 12-month period.
Purpose: To estimate the costs of implementing each evaluated program and to estimate the cost-effectiveness of the programs. |
Mode: The project team asks program leaders for their accounting records or financial reports and obtain as much information as possible from these records. If additional information is needed after review of financial records, the project team asks the programs to complete the workbook in part or in full, depending on the information required.
Duration: 32 hours |
Other Data Sources and Uses of Information
The NextGen Project examines administrative records data for outcomes of interest; this information is already collected by programs and represents no additional burden for participants or program staff. The project team will gather administrative data on quarterly earnings, receipt of unemployment insurance, and new hires on all study participants from the National Directory of New Hires (NDNH), which is maintained by the Office of Child Support Enforcement at ACF. If applicable, the project team will also examine available records for study participants on the receipt of TANF from program data and contact information from state or local TANF agencies. For all programs participating in impact studies, the research team will examine administrative data from SSA on annual taxable earnings and receipt of SSI and Social Security Disability Insurance. In addition, as applicable and informative to the programs’ theories of change, available data might also be examined on receipt of Supplemental Nutrition Assistance Program (SNAP) benefits and contact information; receipt of benefits and contact information from the Special Supplemental Nutrition Program for Women, Infants, and Children; state records on child support owed or paid; health care outcomes (Medicare enrollment and claims) from the Centers for Medicare & Medicaid Services; involvement with the criminal justice system from court records; educational attainment and completion from school districts; and receipt of housing benefits (such as participation in a housing choice voucher program) from housing authorities. The project is also using information collected or expected to be collected under the generic clearance for Formative Data Collections for ACF Research (OMB #0970-0356), including information collected to gather feedback from stakeholders, identify sites, and assess activities and characteristics.
A3. Use of Information Technology to Reduce Burden
This project is using multiple applications of information technology to reduce burden. As described below, information technology is being used to collect baseline data, participant identifying and contact information, and information on service receipt. It will also be used to conduct the two follow-up surveys and collect cost information from the programs. The semi-structured staff discussions and in-depth participant interviews will be audio recorded, if respondents consent to being recorded. Additionally, interviews may be conducted via telephone or video dependent upon any COVID-related restrictions.
RAPTER®. RAPTER® is a secure, web-based system that program staff use to administer consent to participants, collect their identifying and contact information, conduct random assignment, and enter information on the services received or activities participated in by study participants. The use of check boxes and drop-down menus and response categories minimize data entry burden.
Baseline and follow-up surveys. All surveys have the capability to be hosted on the Internet via a live secure web-link. To reduce burden, the surveys employ (1) secure log-ins and passwords so respondents can save and complete the survey in multiple sessions, (2) drop-down response categories so respondents can quickly select from a list, (3) dynamic questions and automated skip patterns so respondents only see those questions that apply to them (including those based on answers provided previously in the survey), and (4) logical rules for responses so respondents’ answers are restricted to those intended by the question.
Respondents also have the option to complete the baseline survey and first and second follow-up surveys using computer-assisted telephone interviewing (CATI). CATI reduces respondent burden, relative to interviewing via telephone without a computer, by automating skip logic and question adaptations and by eliminating delays caused when interviewers must determine the next question to ask.
Excel-based workbook for collecting cost data. A Microsoft Excel-based data collection tool is used to collect cost data. To reduce respondent burden, the project team asks program leaders for their accounting records or financial reports and obtain as much information as possible from these records to complete the workbook. If additional information is needed after review of financial records, the project team asks the programs to complete the remaining sections of the workbook. Formatting, data checks, and layout built into the template assist staff in completing it.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
Information that is already available from alternative data sources will not be collected again for this project. For example, if a program in the study has an existing management information system that collects information needed for this project that is exportable and of sufficient quality, the project team accepts data from its existing system. In these cases, the project team requests that the program only enter into RAPTER® data that the program is not already collecting.
Although information on employment is obtained from administrative records and via the survey, this information is not duplicative because the two sources differ in accuracy and coverage of jobs. NDNH administrative records provide information on quarterly earnings from jobs covered by unemployment insurance as well as new hires. The baseline survey and follow-up surveys ask for information about all jobs held, including those not covered by unemployment insurance. The follow-up surveys collect information about the characteristics of the jobs (such as the wage rate, hours worked, and benefits offered) that are not included in the NDNH data.
The follow-up surveys collect information on whether participants received assistance from public assistance programs such as TANF, SNAP, unemployment insurance, and other assistance programs. However, these surveys do not ask for details about the receipt of these benefits, which we obtain via administrative records. It is important to ask about receipt of benefits on the survey because administrative records are not available for those respondents who do not provide their Social Security number. The study team will, with this nonsubstantive change request, collect Social Security numbers on the follow-up surveys for those participants who did not provide them during enrollment.
As noted in Section A2, the NextGen Project is coordinating with OPRE’s BEES study. OPRE is intentionally and strategically coordinating these projects to prevent duplication of effort; fully capitalize on the opportunity the projects afford for large-scale, rigorous evaluation; advance the knowledge base regarding effective employment strategies for low-income, vulnerable populations; and meet SSA’s priorities across both projects. The projects intentionally included some common questions within instruments. Areas of measurement coordination with the existing BEES data collection instruments are described in the question-by-question justifications for the baseline data collection and follow-up surveys (Appendices B, C, and D, revised). The projects differ in that BEES is especially interested in evaluating programs for individuals and families struggling with opioid use disorders, use of other substances, and/or disability or mental health challenges, while the NextGen Project is especially focused on evaluating interventions that are market-oriented and/or employer-driven.
A5. Impact on Small Businesses
Small organizations, such as businesses or nonprofit organizations, are involved in implementing some of the programs in this study. The project team minimizes the burden for respondents by collecting data at times convenient for the respondents and requiring minimal record keeping or written responses on the part of respondents.
A6. Consequences of Less Frequent Collection
The project team collects information only once for the baseline survey and identifying participant information, staff characteristics survey, program leadership survey, semi-structured staff discussions, semi-structured employer discussions, in-depth participant interviews, and the Excel-based workbook for collecting cost data.
The project team administers two similar follow-up surveys. Collecting data at two points of time allows an examination of whether the impacts of the program changed over time and whether changes in intermediary outcomes (such as health or skills) were associated with changes in longer-term outcomes (namely employment and economic independence outcomes). This also reduces the chance of recall error from respondents when collecting information on their receipt of services and jobs held over a period of time, relative to collecting it only once at the end of the follow-up period. Similarly, updated contact information is collected from respondents upon administration of the first follow-up survey to assist in locating them for the second follow-up survey.
Program staff use the RAPTER® system or their existing management information system to record service receipt for each participant each time he or she receives a service. Staff are asked to enter the information into RAPTER® immediately after the service is provided. Doing so less frequently would contribute to recall error and affect the quality of data collected.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 20, 2022, Volume 87, Number 243, page 77847, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.
Experts in their respective fields from OPRE and Mathematica were consulted in developing the design, data collection plan, and instruments for which clearance is requested. Select agency staff within SSA and HHS were also consulted. The project team also consulted with the BEES project staff to coordinate measurement of key outcomes across projects.
A9. Tokens of Appreciation
The following text describes the tokens of appreciation that were approved and are currently in use; this nonsubstantive change request does not propose any changes.
The structure of tokens of appreciation for this study is designed to support the retention of respondents over the course of the longitudinal data collection and enhance the quality of information. OMB approved the initial proposed structure of tokens of appreciation for this study in April 2020 and approved of changes to those tokens in June of 2022.
Study Enrollment
After finishing the study enrollment process, participants receive a study packet designed to establish their engagement with the study. This packet includes a copy of the consent form, a one-page study flyer that describes upcoming data collection activities (see Appendix G), and a small study-specific item (valued between $1-$3) such as a magnet, keychain, or screen cleaner, that contains the study logo and contact information for our call center. The purpose of these materials is to establish positive association with the study and support familiarity when respondents are contacted to participate in a survey.
Longitudinal Surveys
As agreed upon with OMB, the project team is currently testing a $5 prepaid token of appreciation offered before the sample member responds to the first follow-up survey. To evaluate the effectiveness of the prepaid token using an experimental design, one research group is offered a $5 prepaid token and a $50 postpaid token and the other research group is offered no prepaid token of appreciation but is offered a $55 postpaid token. If the experiment shows the prepaid token is effective for the first follow-up survey, the study team will offer all sample members the $5 prepaid token and a $50 postpaid token of appreciation. If the experiment shows that it is not effective, the study team will convert to the original plan approved by OMB and offer only a $50 postpaid token of appreciation and no prepaid token.
For the second follow-up survey, the project team will use a $5 prepaid token of appreciation if the experiment shows it is effective for the first follow-up survey. Otherwise, second follow-up survey respondents will receive a $50 postpaid token, as approved.
The project team will provide a memorandum to OMB with the experimental results from the first follow-up survey, as described above, and a plan for any changes to the second follow-up survey tokens of appreciation as a result. The project team expects to have the memorandum ready for OMB in 2024.
In-depth Interviews
Respondents to the in-depth participant interviews receive a $60 token of appreciation upon completion of the interview.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information
Information is not maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier. The information provided by or about participants during the baseline data collection, follow-up surveys, and service receipt tracking contains participant-level personally identifiable information (PII). This includes names, addresses, email addresses, social media accounts, phone numbers, birth dates, and Social Security numbers. This information is needed to ensure that: the prospective study participant has not already enrolled in the study; the project team can locate study participants to complete the follow-up surveys; and the project team can link participants to their corresponding administrative data. See Section A11 for further details.
Mathematica will share study participants’ information with SSA, which will do additional research on how programs affect earnings and receipt of disability benefits. They will conclude any such research by 2040. Mathematica will share information such as name, sex, date of birth, and Social Security number so researchers at SSA can locate participants’ records. They will only use this information to do research. The information will not be used to make decisions about benefits participants receive from the SSA, now or in the future. The sharing of information with SSA for these purposes and for the specified timeframe are described to participants in the informed consent form (Appendix A).
Assurances of Privacy
Information collected is kept private to the extent permitted by law. Respondents are informed of all planned uses of data, that their participation is voluntary, and that their information is kept private to the extent permitted by law. As specified in the contract, Mathematica complies with all Federal and Departmental regulations for private information.
Due to the sensitive nature of this research (see A.11 for more information), the evaluation obtained a Certificate of Confidentiality (Appendix R). The Certificate of Confidentiality helps to assure participants that their information is kept private to the fullest extent permitted by law. The project team also secured Institutional Review Board (IRB) approval from the Health Media Lab IRB for all impact, descriptive, and cost studies.
Data Security and Monitoring
As specified in its contract with OPRE, Mathematica protects respondent privacy to the extent permitted by law and complies with all Federal and Departmental regulations for private information. Mathematica developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. Mathematica ensures that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. All study staff with access to PII—including program staff who are entering information about study participants and their service receipt into RAPTER®—receive study-specific training on (1) limitations on disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. These procedures are documented in training manuals for study staff, and refresher trainings will occur annually.
Mathematica uses Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. Mathematica securely generates and manages encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. Mathematica ensures that this standard is incorporated into their property management/control system and has procedures to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically is secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, Mathematica’s Data Safety and Monitoring plan minimizes to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.
Information shared with researchers at SSA (see discussion above) and exchanged between programs and Mathematica is sent via a secure file transfer protocol.
At the end of the study, de-identified project data will be archived to make them available to other researchers. Mathematica is working with ACF to develop a comprehensive data archive plan; later, Mathematica will work with ACF to produce archive data files. Any restricted- or public-use files will be reviewed for appropriateness of public or restricted release, including appropriate masking techniques for each level of release. A non-disclosure review will also be conducted to ensure that the data cannot be used to re-identify study participants.
A11. Sensitive Information
To evaluate the effectiveness of employment programs for vulnerable populations, it is necessary to ask some sensitive questions. Before starting the baseline and follow-up surveys and in-depth interviews, all respondents are informed that their identities are kept private to the extent permitted by law, that results will only be reported in the aggregate, that their responses will not affect any services or benefits they or their family members receive, and that they do not have to answer any questions that make them uncomfortable.
The sensitive questions in the approved data collection instruments for this ICR follow. These topics were all described in previously submitted and approved justification packages.
Respondents’ Social Security numbers. Respondents’ Social Security numbers are necessary to collect administrative data used to estimate impacts on earnings, employment, and public benefit receipt. The consent form informs study participants that the project team might collect administrative data about them. Social Security numbers will also be used to collect information through online databases containing information on the location of study participants for the follow-up surveys. Along with names, birthdates, and other data from baseline surveys, Social Security numbers will be used to verify respondents’ identities for follow-up surveys. The project team does not want to rely on name and address matching (or similar techniques) for collecting administrative data because it leads to the inability to match administrative data for a high proportion of participants, an unacceptably high uncertainty in match success, or both. This would affect the study’s ability to estimate impacts and draw conclusions for findings that rely on administrative data. This nonsubstantive change request seeks to collect Social Security numbers via the two follow-up surveys if the numbers were not provided during enrollment.
Wage rates and earnings. It is necessary to ask about earnings because increasing participants’ earnings is a key goal of these programs. The follow-up surveys ask about each job worked since random assignment, the wage rate, and the number of hours worked per week. This information is collected on the first and second follow-up surveys.
Challenges to employment. We ask about some challenges to employment caused by COVID-19. This provides some information about the labor market context of the participants at the time they enroll in the study.
Economic hardships. The follow-up surveys ask about economic hardships, such as food insecurity. These outcomes are used to assess respondents’ degree of economic independence and might be affected by the program. Economic hardships might also be discussed as part of the in-depth participant interviews.
Disabilities, mental and physical health, and substance misuse. The baseline and follow-up surveys collect information about disabilities, mental or other health problems, and substance misuse; the severity of those issues; and how much they impact the ability to work. These issues might also be discussed in the in-depth participant interviews. All of these are important potential challenges to finding or maintaining employment and could play a role in the effectiveness of the program. The Center for Epidemiologic Studies Depression Scale Revised (CESD-R) is also collected for one program during eligibility screening with the data saved for those determined to be eligible for the program and who consent to participate in the study.
Involvement in the criminal justice system. The baseline survey asks about prior involvement in the criminal justice system, including the number of convictions and felony convictions, details about parole or probation, type of crime committed, and time spent in last incarceration because such involvement often makes it harder to find employment. The two follow-up surveys also ask about arrests, convictions, and incarcerations that occurred after random assignment because these outcomes might be affected by the program. Criminal history might also be discussed during the in-depth participant interviews.
COVID-19-related challenges. The baseline survey asks if respondents are fully vaccinated against the Coronavirus because vaccination is expected to be associated with employment outcomes. It also asks whether COVID-19 posed specific challenges to employment for study participants or if the pandemic impacted previous employment. The follow-up surveys ask some questions about the effects of the COVID-19 pandemic on getting or keeping employment and whether they are vaccinated against the Coronavirus.
A12. Burden
Explanation of Burden Estimates
This nonsubstantive change request does not change the average burden per response for any of the data collections. The number of respondents reflects the estimated number of respondents over the three-year extension period that started in April 2023. Overall, the project does not expect to exceed the previously approved annual burden of 9,241 hours. Table A.3 reflects the burden for information collection over the three-year extension period.
Details of the estimates for data collections in this request are as follows:
Baseline data collection. Baseline data collection involves both study participants and program staff. The burden estimates reflect that program staff assist study participants in baseline data collection, which includes collecting the baseline survey (Instrument 1) and using RAPTER® to collect participant identifying and contact information (Instrument 2). For the NextGen Project in total, we expect about 4,200 study participants will complete baseline data collection.
We expect about 3,000 study participants will complete baseline data collection over the three-year extension period. Baseline data collection (inclusive of the baseline survey and RAPTER® identifying and contact information) takes an average of 0.42 hours, for a total of 1,260 burden hours. Annualizing over three years is 420 hours per year for study participants.
We assume that 120 program staff will perform the baseline data collection. Each staff member will administer the baseline data collection (inclusive of the baseline survey and RAPTER® identifying and contact information) 25 times and each session is expected to last 0.42 hours for a total of 1,260 burden hours. Annualizing over three years is 420 hours for program staff.
First and second follow-up surveys. In total for the NextGen Project, we expect to survey 4,200 study participants at two follow-up points. We anticipate an 80 percent response rate or 3,360 respondents to each survey and we expect each survey to last an average of 50 minutes (0.83 hours). The addition of the Social Security question to each survey as part of this nonsubstantive change request will not impact the average survey burden. During the extension period, we anticipate 3,100 respondents to the first follow-up survey for a total of 2,573 burden hours. Annualizing over three years is 858 per year for study participants. We anticipate all 3,360 responses to the second follow-up survey to occur during the extension period, for a total of 2,789 burden hours. Annualizing over three years is 930 hours per year for study participants.
Service receipt tracking. In total, we anticipate 80 program staff will enter data on program service receipt into RAPTER® and will enter 250 service receipt records. Over the three-year extension period, we expect 80 staff to make 150 entries per staff member and expect that each entry will take 5 minutes (0.08 hours), or a total of 960 burden hours. Annualizing over three years is 320 hours.
Staff characteristics survey. In total, we expect to survey 120 program staff who directly interact with participants. Over the three-year extension period we expect to survey 20 program staff. The survey is expected to take 25 minutes (0.42 hours) to complete, or a total of 8 hours. Annualizing over three years is 3 hours.
Program leadership survey. In total for the NextGen Project, we expect to survey 30 program leaders. Over the three-year extension period we expect to survey 5 program leaders. The survey is expected to last 15 minutes (0.25 hours) to complete, or a total of 1 burden hours. Annualizing over three years is 1 hour.
Semi-structured program discussion guide—program leaders. In total for the NextGen Project, we expect to interview 24 program leaders. Over the three-year extension period we expect to interview 4 program leaders. We expect each staff interview to last 1.5 hours on average, or a total of 6 burden hours. Annualizing over three years is 2 hours.
Semi-structured program discussion guide—program supervisors and partners. In total for the NextGen Project, we expect to interview 48 program supervisors and partners. Over the three-year extension period we expect to interview 8 program supervisors or partners. We expect each interview to last one hour on average, or a total of 8 burden hours. Annualizing over three years is 3 hours.
Semi-structured program discussion guide—program staff and providers. In total for the NextGen Project, we expect to interview 48 direct service staff at programs and providers. Over the three-year extension period we expect to interview 8 direct service staff. We expect each staff interview to last one hour on average, or a total of 8 burden hours. Annualizing over three years is 3 hours.
Semi-structured program discussion guide—employers. In total for the NextGen Project, we expect to interview 30 employers’ staff. Over the three-year extension period we expect to interview 8 employers’ staff. We expect each interview to last one hour on average, or a total of 8 burden hours. Annualizing over three years is 3 hours.
In-depth participant interview guide. In total for the NextGen Project, we expect to interview 120 study participants. Over the three-year extension period we expect to interview 20 study participants. These interviews are expected to last two hours on average, or a total of 40 burden hours. Annualizing over three years is 13 hours.
Cost workbook. In total for the NextGen Project, we expect that 28 program staff will enter data on expenditures and costs into Excel. Over the three-year extension period, we expect that 24 program staff will enter data on expenditures and costs into Excel. We expect one entry per staff member and expect that each entry will take 32 hours, or a total of 768 burden hours. Annualizing over three years is 256 annual burden hours.
Estimated Annualized Cost to Respondents
The total annual cost to respondents over the three-year extension period is estimated to be $36,033. The estimated cost figures are computed from the total annual burden hours and an average hourly wage for staff and participants. The wage rate for program staff administering the survey is based on the May 2021 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_stru.htm).
The rate used for direct service staff, $19.45, is the mean wage for social and human services assistants under SOC code 21-1093.
The rate used for program leaders is $55.08, the mean hourly wage of local government general and operation managers under SOC code 11-1021.
The rate used for program and partner supervisors is $36.92, the mean hourly wage for social and community services managers under SOC code 11-9151.
The average hourly wage for employers is estimated as $55.41, the average hourly wage of general and operations managers across industries under SOC code 11-1021.
The average hourly wage of study participants is estimated to be $7.25, the federal minimum wage.
Table A.3 reflects the estimated cost for information collection over the three-year extension period.
Table A.3. Burden and cost for information collection extension
Instrument |
No. of Respondents (total over request period) |
No. of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Annual Burden (in hours) |
Average Hourly Wage Rate |
Total Annual Respondent Cost |
Baseline survey & Identifying and contact information — participants |
3,000 |
1 |
0.42 |
1,260 |
420 |
$7.25 |
$3,045 |
Baseline survey & Identifying and contact information — staff |
120 |
25 |
0.42 |
1,260 |
420 |
$19.45 |
$8,169 |
First follow-up survey—participants |
3,100
|
1 |
0.83 |
2,573 |
858 |
$7.25 |
$6,221 |
Second follow-up survey—participants |
3,360 |
1 |
0.83 |
2,789 |
930 |
$7.25 |
$6,743 |
Service receipt tracking—program staff |
80 |
150 |
0.08 |
960 |
320 |
$19.45 |
$6,224 |
Staff characteristics survey — staff |
20 |
1 |
0.42 |
8 |
3 |
$19.45 |
$58 |
Program leadership survey — program leaders |
5 |
1 |
0.25 |
1 |
1 |
$55.08 |
$55 |
Semi-structured program discussion guide —program leaders |
4 |
1 |
1.50 |
6 |
2 |
$55.08 |
$110 |
Semi-structured program discussion guide —program supervisors and partners |
8 |
1 |
1.00 |
8 |
3 |
$36.92 |
$111 |
Semi-structured program discussion guide —program staff and providers |
8 |
1 |
1.00 |
8 |
3 |
$19.45 |
$58 |
Semi-structured program discussion guide —employers |
8 |
1 |
1.00 |
8 |
3 |
$55.41 |
$166 |
In-depth participant interviews — participants |
20 |
1 |
2.00 |
40 |
13 |
$7.25 |
$94 |
Cost workbook—program staff |
24 |
1 |
32.0 |
768 |
256 |
$19.45 |
$4,979 |
Totals |
3,232 |
|
$36,033 |
||||
A13. Costs
There are no additional costs to respondents.
A14. Estimated Annualized Costs to the Federal Government
The total cost to the Federal government for all data collection activities under this OMB number will be about $15,548,000. Annualized costs to the Federal government will be about $5,182,667 for the data collection. These estimates of costs are derived from Mathematica’s budgeted estimates and include labor rates, direct costs, and tokens of appreciation for respondents.
Activity |
Detail |
Estimated Cost |
Survey administration |
|
$9,449,000 |
Analysis and initial dissemination |
|
$6,099,000 |
Total costs over the request period |
$15,548,000 |
|
Annual costs |
$5,182,667 |
|
A15. Reasons for changes in burden
The requested changes submitted as part of this nonsubstantive change request do not change the burden estimates for the extension period that started in April 2023.
A16. Timeline
The beginning of participant intake and baseline data collection is staggered by program. Due to delays in the study schedule due to COVID-19, the first programs began baseline data collection in 2021 and later programs in 2022. For each program, we expect intake and baseline data collection to continue for about 18 to 30 months. Data collection for the descriptive and cost studies began in 2021 or 2022 depending on the program. The first follow-up survey began in 2022 and the second follow-up survey began in 2023.
Findings from the project will be published throughout the study in technical reports and briefs. We anticipate that reporting on the descriptive and cost studies continue through 2024. Reporting on the intermediate impact findings will likely begin in 2026 and continue through 2027. Reporting on final impact findings will likely begin in 2027 and continue through 2028.
We anticipate that data archives (restricted or public use) would become available starting in 2029 and hosted on a data archive platform such as the Inter-university Consortium for Political and Social Research (ICPSR).
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Instruments
Instrument 1. Baseline survey – revised (approved June 2022)
Instrument 2. Identifying and contact information – revised
Instrument 3. First follow-up survey – revised August 2024
Instrument 4. Second follow-up survey – revised August 2024
Instrument 5. Service receipt tracking – revised
Instrument 6. Staff characteristics survey – revised
Instrument 7. Program leadership survey – revised
Instrument 8. Semi-structured program discussion guide – revised
Instrument 9. Semi-structured employer discussion guide – revised
Instrument 10. In-depth participant interview guide – revised
Instrument 11. Cost workbook
Appendices
Appendix A. Informed consent form – revised
Appendix A.1. Bridges consent forms
Appendix B. Question-by-question justification for baseline survey – revised (approved June 2022)
Appendix C. Question-by-question justification for identifying and contact information – revised
Appendix D. Question-by-question justification for follow-up surveys – revised August 2024
Appendix G. Follow-up survey reminders and notifications – revised
Appendix G.1. NextGen Project recruitment materials
Appendix P. Federal Register Notice
Appendix P.1. Federal Register Notice – 30-day request, published January 2021
Appendix P.2. Federal Register Notice – 30-day request, published March 2022
Appendix P.3. Federal Register Notice – 60-day request, published December 2022
Appendix Q. Summary of requested changes (submitted February 2021)
Appendix Q.1. Summary of requested changes – revised (approved June 2022)
Appendix R. Certificate of Confidentiality
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Newell, Gabrielle (ACF) |
| File Modified | 0000-00-00 |
| File Created | 2024-09-06 |