NonSub Change Memo - NextGen Project_Nov 2020

MEMORANDUM

To: Jordan Cohen, Office of Information and Regulatory Affairs, Office of Management and Budget (OMB)

From: Hilary Bruck, Gabrielle Newell, and Marie Lawrence

Office of Planning, Research, and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: November 12, 2020

Subject: Non-substantive Change Request – Next Generation of Enhanced Employment Strategies Project [Impact, Descriptive, and Cost Studies] (OMB # 0970-0545)

Background

Type of Request: This memo requests approval of non-substantive changes to data collection instruments for the Next Generation of Enhanced Employment Strategies (NextGen) Project. Specifically, we seek clearance for non-substantive changes to the following:

• Updates to the previously approved Phase 1 instruments (OMB #0970-0545);

• Updates to the previously approved consent form and clearance for additional versions of the consent form tailored for one of the programs selected for the evaluation; and

• Approval to use a subset of Phase 2 instruments with programs selected for inclusion in the project, with non-substantive changes made to those instruments.

These requested non-substantive changes do not require any changes to the burden estimates.

Study Features Salient to Request: The NextGen Project is intended to build on the findings and lessons learned from past and ongoing evaluations by identifying and rigorously evaluating the “next generation” of employment strategies for highly vulnerable populations with complex barriers to obtaining and retaining employment. The project includes impact, descriptive, and cost studies of up to 10 interventions.

In April 2020, OMB approved the new information collection request (ICR) for Phase 1 data collection instruments for the NextGen Project. The first phase submission included instruments that will be uniform across programs selected for inclusion in the NextGen Project and participating in impact evaluations. These include the informed consent form (Appendix A), the baseline survey (Instrument 1), and identifying and contact information (Instrument 2). In the first ICR, we included drafts of Phase 2 data collection instruments and associated burden estimates for initial review, information purposes, and public comment. We indicated that some of the Phase 2 instruments may require revisions to tailor to each program selected for the evaluation and that, as programs are selected, we will submit tailored Phase 2 instruments—as needed—as either a non-substantive change request or a revision with abbreviated public comment time, dependent on the level of changes and guidance provided by OMB.

As the NextGen Project has engaged in assessing promising programs for inclusion in the study (with approval for these activities granted under the generic clearance for Formative Data Collections for ACF Research, OMB #0970-0356), we have gained knowledge that suggests necessary refinements to the Phase 1 instruments. In response to this knowledge as well as the 2019 novel coronavirus disease (COVID-19) pandemic, we are seeking non-substantive changes to the Phase 1 data collection instruments (Instrument 1. Baseline survey - revised and Instrument 2. Identifying and contact information - revised) as well as the study’s informed consent form (Appendix A. Informed consent form - revised). Additionally, we are seeking clearance for tailored versions of the consent form for use by one program selected to participate in the evaluation (Appendix A.1. Bridges consent forms). We anticipate that this program will be the only one in the evaluation that will enroll youth and therefore requires parent/guardian consent forms and youth assent forms.

We are also seeking official approval to use a subset of the Phase 2 instruments, with non-substantive changes to all but one of these instruments, across all selected NextGen Project sites. Rather than tailoring instruments to each selected site, as proposed in the first ICR, we intend to use the same Phase 2 instruments across all sites, with skip patterns and instructions to interviewers indicating whether certain items only apply to certain types of respondents or programs. The non-substantive changes proposed to the Phase 2 instruments include revisions to capture how programs responded to COVID-19 and the resulting recession. Below are additional details regarding these requests.

Time Sensitivity: We expect to first use Phase 1 data collection instruments in April 2021. However, we need OMB clearance for these instruments by November 2020 so that we have time to obtain the necessary approvals for conducting research from local institutional review boards. Additionally, we are planning to begin descriptive and cost data collection in some selected sites in late 2020, to capture information on the effects of COVID-19.

REQUEST DETAILS: PHASE 1 INSTRUMENTS

Justification

The non-substantive changes requested for the Phase 1 instruments are critical to understand the impact of the programs in the evaluation. With these changes, the study will be more likely to reach participants for later follow-up data collections, will be better able to link participants to program data to better understand service receipt, and will be able to collect more relevant baseline data for the program evaluations. Details about requested changes follow.

Non-Substantive Change Details

Informed Consent Form (Appendix A. Informed consent form - revised)

We propose the following non-substantive changes to the study’s approved informed consent form:

• Language to allow study programs to share additional information about participants (their contact information and referral source) with the study team. This information is useful for locating study participants for later data collections and for understanding how participants are referred to programs for assistance.

• Language to cover the collection of participants’ contact information from public assistance programs to facilitate locating study participants for later data collections.

• Revision to cover collecting administrative data for up to 20 years after study enrollment, rather than for up to 10 years, to allow for estimation of longer-term impacts on participants for research purposes.

• Revision to the date for record retention for the Social Security Administration (SSA) to conduct research on study participants from 2028 to 2040. This reflects the current and expected delays in the study schedule due to COVID-19.

• Removal of the parent/guardian consent portion of the consent form. As noted above, we anticipate that only one program selected for the evaluation—the Bridges from School to Work program—will enroll youth. We have therefore developed separate versions of the consent form tailored to this program—one to collect consent from parents/guardians of youth; one to collect assent from youth; and one to collect consent from participants age 18 and older. These consent form versions are included with this request in Appendix A.1. Bridges consent forms.

• Revisions to include language regarding consent for recording of study interviews. Added in the event that other programs selected for the evaluation in the future work with school districts; this language will only appear if a program works with school districts and the applicant attends school in a district that requires consent for recording of the study interviews.

• Minor revisions requested by the project’s institutional review board (IRB):

  • Revisions to clarify legal situations when researchers might share confidential information with authorities under a Certificate of Confidentiality

  • Removal of a sentence stating that the IRB may inspect confidential study information to ensure protection of participants’ rights

  • Revisions to lower the overall reading level for the consent form

Baseline Survey (Instrument 1. Baseline survey - revised)

We propose the following non-substantive changes to the baseline survey. These proposed changes do not impact the burden estimates provided in the original ICR submission, given site-specific skip patterns within the survey.

• We made minor changes to the response options to the youth-based housing situation question (B9a) to collect information on whether youths pay rent to their parents or guardians.

• We revised the wording for the household benefit question (B13 and B14) to strike the mention of “or anyone in your household” for respondents living in a halfway house or group home. This change will help simplify the question for affected respondents and allow the study team to collect more accurate information about respondents’ households.

• We revised the criteria for who will be asked four items about the receipt of SSA benefits (B14, B15, B16, and B17). These items will now be asked of respondents from a potential program serving the SSA target population only if the program does not already use these questions to screen for SSA benefits as part of program eligibility. This change will reduce duplication of information collection and streamline the survey for such a program.

• To better measure social support outcomes, we modified the question wording and response options for B18 (emergency support) and B19 (financial emergency support). Asking how many people someone can turn to for help, as opposed to just if they have someone to turn to, will allow a better baseline estimate of social supports for emergencies or need.

• We added three questions, B20, B21, and B22, on social trust. These questions will be asked of applicants to potential programs that are focused on building trust.

• Some programs being considered for the NextGen Project serve individuals recently released from jail or prison. We made the following changes to make specific items more appropriate for this population or to collect data on variables that may predict employment outcomes:

  • The word “current” was added to item B9 (type of housing) to ensure that we collect information about their housing following release from jail or prison.

  • We changed the structure of the past employment questions (C1 to C4a) so that they ask about employment prior to their last arrest. Employment prior to incarceration has been found to be a strong predictor of employment after incarceration.

  • Item C8 (ever been arrested) would not need to be asked since all participants would have a prior arrest; we therefore added a skip pattern and adjusted the wording slightly in C8 for all other programs.

  • C9 was revised to collect number of convictions rather than whether ever convicted. A new item, C9a, was added to collect the number of felony convictions.

  • Question wording and response options for item C10 were revised to collect more detail about parole or probation.

  • Item C10a was added to collect the type of crime committed and C12a was added to collect time spent in last incarceration.

• We revised C5 and C6 to only focus on challenges to working, instead of including training and education. We added item C6b1 to collect information on whether not having documents needed for work was a specific challenge to employment.

• We added some items (C4b, C5a, C6e1, C6e2, C6r, and C6s) to collect information on whether COVID-19 posed specific challenges to employment for study participants and if COVID-19 impacted prior employment.

We have updated the question-by-question justification for the baseline survey to reflect the changes proposed above, and included it with this non-substantive change request submission (Appendix B. Question-by-question justification for baseline survey - revised).

Identifying and Contact Information (Instrument 2. Identifying and contact information - revised)

RAPTER® is a secure, web-based system that program staff will use to determine program eligibility (for some programs), administer consent to participants, collect their identifying and contact information, and conduct random assignment. Proposed non-substantive changes to the RAPTER identifying and contact information collection (Instrument 2) do not impact the burden estimates provided in the original ICR submission. We propose the following revisions to Instrument 2:

• One program under strong consideration for inclusion in the study currently uses the Center for Epidemiologic Studies Depression Scale Revised (CESD-R, added as item B2) as a screening tool. Although these items are a program requirement to determine eligibility (not part of the NextGen evaluation), we propose to include these questions in RAPTER to facilitate the program’s intake process. Program staff would be able to administer this scale and receive an immediate score indicating the person’s eligibility for the program. If an individual is eligible for the program based on the CESD-R score, staff could then immediately administer the consent form. The study team will maintain CESD-R scores for those who are eligible for the program (score of more than 16 on the scale) and also consent to participate in the study. The consent form indicates that the program may share the eligibility screener score with the study team (Appendix A. Informed consent form – revised).

• The CESD-R will only display for the program that requires it as part of its eligibility screening to facilitate program intake; other programs will skip these items.

• The study is not planning to use consent by proxy so we removed this option in item B3. If interventions involve adults or youths with cognitive disabilities, the project will rely on determinations, screenings, or assessments made by site staff to ensure the potential participants are capable of understanding the consent process and implications of participating in the study. If site staff determine that a potential participant is unable to understand, that individual will be exempt from the study.

• As noted above, we anticipate that only one program selected for the evaluation will enroll youth. The following questions are only applicable to that program and will not display for programs that do not enroll youth. In the event that other programs selected for the evaluation in the future work with school districts and/or youth, the items will display for those programs, as applicable.

  • For item B3, we added a new question to ask if the applicants enrolling in the study are in a prefilled school district name. This item will allow us to find out if enrollees are part of a school district for interventions that may work with school districts. It will only be displayed for interventions that take place in schools.

  • Also for item B3, we added a question to ask if the applicant consents to being recorded. This question will only be asked if the applicant attends school in a district that requires consent for recording of the study interviews. This aligns with the revisions made to the consent forms (Appendix A. Informed consent form – revised; and Appendix A.1. Bridges consent forms), as described above.

  • We added the same two items to B4, consent by parent/guardian, as B3. B4 now asks if applicants are part of a school district and asks for parental/guardian consent for recording of study interviews. This aligns with the parent/guardian consent form tailored for the Bridges program (Appendix A.1. Bridges consent forms).

• We added a question to item B5 to collect participants’ Instagram addresses. OMB approved the collection of Instagram addresses under the Evaluation of Employment Coaching for TANF and Other Related Populations (OMB #0970-0506) and it has been helpful for locating study participants for later data collections.

• We added a question to item B5 to collect the program’s client identification number. This will allow the research contractor to share data with the program using only identification numbers rather than personally identifiable information.

• We added one question (item B9) about the likelihood that the study participant will be assigned to a career navigator. This item will be asked only for programs in which the study participant may be assigned to a career navigator, and will be used to estimate the impact of the program on the subgroup of study participants who are likely to be assigned to a career navigator.

We have updated the question-by-question justification for the identifying and contact information to reflect the changes proposed above and included it with this non-substantive change request (Appendix C. Question-by-question justification for identifying and contact information - revised).

REQUEST DETAILS: Phase 2 Instruments

We are currently seeking official approval to use a subset of the Phase 2 instruments across all selected NextGen Project sites, with non-substantive changes to all but one of these instruments (described further below). The original ICR submission included burden estimates for each Phase 2 instrument. The burden for the instruments included in this request falls within those original estimates; the proposed non-substantive changes do not change the burden estimates. We will seek approval to administer the remainder of the Phase 2 instruments in a future ICR. The following instruments are part of this request:

1. Instrument 6. Staff characteristics survey - revised

2. Instrument 7. Program leadership survey - revised

3. Instrument 8. Semi-structured program discussion guide - revised

4. Instrument 10. In-depth participant interview guide – revised

5. Instrument 11. Cost workbook

Details about requested changes to Instruments 6, 7, 8, and 10 follow. There are no changes proposed to the Phase 2 Excel-based cost workbook (Instrument 11) to collect cost study data for all NextGen Project sites.

Non-Substantive Change Details

Surveys of Program Staff and Leaders (Instrument 6. Staff characteristics survey – revised; Instrument 7. Program leadership survey – revised)

These changes are necessary to better reflect the activities program staff and leaders might conduct; streamline items where able and appropriate; and capture information about program changes related to COVID-19.

• We added response options to B1 and B2 of Instrument 6 to capture additional staff activities.

• We changed the response options for B4 and B5 of Instrument 6 and B3 of Instrument 7 to make them easier for respondents to complete and to capture more precise information.

• We added a response option to B8 of Instrument 7 to capture additional program leader activities.

• We deleted C1 and C2 of Instrument 6 and C1 of Instrument 7 because we are requesting similar information in Instrument 8. Semi-structured program discussion guide. We revised the Section C. introduction text to reflect removal of these items.

• We added the word “now” to C3 and renumbered it to C1 of Instrument 6. We made the same change to C2 of Instrument C7 and renumbered it to C1.

• We added sub-items to D1 on Instrument 6 and C2 on Instrument 7 to capture the effects of COVID-19 on program operations and clarified that respondents should consider all the statements under this question as of “right now.” We added logic to skip these subitems if the COVID-19 pandemic is no longer applicable at the time of data collection.

Program Discussion Guide (Instrument 8. Semi-structured program discussion guide – revised)

These changes are necessary to understand programs’ responses to contextual factors including the pandemic and recession and the social justice and equity movement.

• We added interviewer notes and probes to capture changes in programs due to COVID-19.

• We made slight changes to E4 to E7 to ask about refining the intervention model in addition to asking about its development.

• We revised wording and added probes at question I2 to make it easier to answer for participants.

• We added question I3 to capture race and equity issues around program administration.

We also made revisions to the mode for this instrument in Supporting Statement A, to indicate that the discussions may be conducted via telephone or video depending on any COVID-related restrictions that prevent in-person discussions.

Participant Interview Guide (Instrument 10. In-depth participant interview guide – revised)

These changes are necessary to understand programs’ responses to the pandemic and recession.

• We made minor revisions to the introductory text to mention COVID-19.

• We made minor revisions to item A1 and added A2 to ask whether respondents are in school. This will help interviewers determine which subsequent questions are appropriate to ask.

• We made minor revisions to item C8c to remove reference to receiving a stipend.

• We added C12 to ask how COVID-19 affected participants’ engagement in the program.

• At the end of section D, we added interviewer notes to confirm respondents’ email or mailing address for the purposes of delivering the interview token of appreciation (a gift card), in cases where the interview is conducted virtually.

We also made revisions to the mode for this instrument in Supporting Statement A, to indicate that the interviews may be conducted via telephone or video depending on any COVID-related restrictions that prevent in-person discussions.