HPOG 2 CovidCohortSurvey_LT-Tracking_SSA_Rev_passback 072721

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants

OMB Information Collection Request

0970 - 0462

Supporting Statement

Part A

July 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Nicole Constance

Lisa Zingman

Amelia Popham

Part A

Executive Summary

1. Type of Request: This Information Collection Request is for a revision to OMB # 0970-0462. We are requesting a 3-year approval.

2. Progress to Date: In 2015, the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF) awarded grants to 32 organizations to administer the second generation of Health Profession Opportunity Grants (HPOG 2.0) Program. ACF contracted with Abt Associates and partners to conduct the HPOG 2.0 National and Tribal Evaluations. Grantees began enrolling participants in 2016 and enrollment will continue through the end of the grant period in September 2021. Through the end of April 2021, the 32 grantees have enrolled over 54,000 participants. Since August 2015, ACF received approval to collect baseline data from HPOG 2.0 National and Tribal Evaluation study participants under OMB # 0970-0462. Since that initial approval, OMB has approved

   • The participant contact update forms;

   • Two follow-up surveys and supporting materials and a phone-based assessment protocol;

   • Several descriptive evaluation protocols; and

   • a program cost survey.

Data collection in support of the Tribal Evaluation is complete. Data collection in support of the National Evaluation is ongoing, including the HPOG Participant Accomplishment and Grantee Evaluation System (PAGES) data collection, Participant Welcome Packets and Contact Update Forms, and the Intermediate Follow-up Survey.

ACF now proposes to begin data collections related to the HPOG 2.0 long-term follow-up study. ACF is also interested in fielding a short-term follow-up survey of HPOG 2.0 participants enrolled after the onset of the COVID-19 pandemic. These two efforts are the focus of this information request.

3. Previous Terms of Clearance: There were no terms of clearance for the most recent approval (November 2020).

4. Summary of changes requested: This is the fourth request for revisions to OMB #0970-0462. This revised request for clearance seeks approval for:

1. COVID-19 Cohort Short-term Survey (COVID-19 Cohort STS): This survey is a revised version of the previously approved Short-term Follow-Up Survey. The goal is to measure the short-term impacts of HPOG 2.0 for study participants enrolled after the onset of the COVID-19 pandemic (May 2020-September 2021). This will help us understand the impact of COVID-19 on program operations and participant outcomes. This also includes revisions to the supplemental materials associated with this data collection.

2. New supporting materials to support the re-engagement of participants selected for a long-term follow-up survey of HPOG 2.0 participants—a newsletter and an outbound call script for collecting updated contact data, both of which will help enhance the contact information update procedures previously approved under OMB #0970-0462.

We do not intend for this information to be used as the principal basis for public policy decisions.

5. Time Sensitivity: The goal is to begin survey data collection for the COVID-19 Cohort in November 2021. We are seeking OMB approval by August 2021 to ensure ample time to program and test the approved instrument. The long-term tracking changes would not take effect until May 2022.

A1. Necessity for Collection

The Health Profession Opportunity Grants (HPOG) Program was authorized by the Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 119, March 23, 2010, sect. 5507(a), “Demonstration Projects to Provide Low-Income Individuals with Opportunities for Education, Training, and Career Advancement to Address Health Professions Workforce Needs,” adding sect. 2008(a) to the Social Security Act, 42 U.S.C. 1397g(a). In 2015, the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF) awarded grants to 32 organizations—27 non-tribal and 5 tribal—to administer the second generation of Health Profession Opportunity Grants (HPOG 2.0) Program. HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects Accordingly, ACF is using a multi-pronged and comprehensive evaluation strategy to understand the effectiveness of HPOG 2.0 programs. Data collection instruments approved under OMB #0970-0462 support this evaluation strategy through:

• collection of uniform performance measures from HPOG programs;

• the National Evaluation, which includes a descriptive evaluation, impact evaluation, and cost-benefit analysis of 27 non-tribal programs; and

• the Tribal Evaluation, which includes implementation and outcomes studies of the five tribal HPOG programs.

Data collection activities began in 2016 when grantees began enrolling participants. Enrollment will continue through the end of the grant period in September 2021. Data collection in support of the Tribal Evaluation is complete. Data collection in support of the National Evaluation is ongoing.

ACF now proposes to begin reengagement activities in anticipation of conducting the HPOG 2.0 long-term follow-up study, which will use survey and administrative data to estimate program impacts at the local and national level and to explore characteristics of local programs that are associated with more favorable outcomes. ACF is also interested in fielding a short-term follow-up survey of HPOG 2.0 participants enrolled after the onset of the COVID-19 pandemic to better understand how the experiences and outcomes of those enrolled after the onset of the COVID-19 pandemic compared to those who enrolled pre-pandemic.

A2. Purpose

Purpose and Use

HPOG provides healthcare occupational training for Temporary Assistance for Needy Families (TANF) recipients and other low-income people to train for jobs that pay well and are expected to be in demand. OMB has approved various data collection activities in support of the HPOG 2.0 National and Tribal Evaluations under OMB #0970-0462 (see Study Design below for more information about the timeline). The evaluation of non-tribal programs will assess the implementation and impacts of HPOG in non-tribal HPOG programs and will include a cost-benefit analysis. Key participant outcomes of interest include (but are not limited to) educational progress, employment, and earnings.

By extending data collection to include a long-term survey, OPRE can address important unanswered questions for policymakers and practitioners. The HPOG 2.0 long-term survey will provide insights into the long-term impacts of HPOG 2.0 for outcomes that are not captured in administrative records, such as details about educational experiences, characteristics of employment, self-employment and earnings from jobs not covered in administrative data, receipt of public assistance, physical and mental well-being, and child outcomes. Survey-based outcomes have been critical to understanding the evolution of outcomes and impacts in other Career Pathways evaluations, specifically, the evaluations of Pathways for Advancing Careers and Education (PACE) and first round of HPOG (HPOG 1.0). The long-term follow-up survey instrument and supporting materials will be part of a later information collection request. This information collection request seeks approval for minor changes for the previously approved contact update procedures (a script for phone administration of Instrument 5b HPOG 2.0 Contact Update Form Phone Script, previously approved in June 2017 and new supplemental material (Attachment AA HPOG2.0 Participant Newsletter).

As a result of the recent COVID-19 outbreak, most HPOG programs adapted their programs in numerous ways—reduced class sizes, shifted to virtual classes, suspended training temporarily, relaxed certification requirements or hands-on coursework, etc. The employment landscape also changed for participants enrolling and participating in healthcare training programs after the onset of the COVID-19 pandemic, relative to those trained prior to the pandemic. In order to understand the effect of the pandemic on the HPOG 2.0 Program overall, it will be important to understand the experiences of those enrolled after the onset of the COVID-19 pandemic compared to those who enrolled pre-pandemic. ACF is considering conducting a survey with a cohort of study participants enrolled after the onset of the COVID-19 pandemic to better understand how the pandemic affected the program and outcomes for participants. This information collection request seeks approval of Instrument #12a, COVID-19 Cohort Short-term Follow-up Survey (COVID-19 Cohort STS) as well as the supplemental materials associated with this data collection (Attachment K-Revised_ COVID-19 Cohort Short-Term Survey Advance Letter, and Attachment M-Revised COVID-19 Cohort Survey Trying to Reach You Flyer, Attachment N-Revised COVID-19 Cohort Short Term Survey Email Reminder Text).

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

Exhibit A-1 summarizes the research questions for this information collection request. Attachment O lists research question associated with the previously approved HPOG 2.0 National and Tribal Evaluation information collections.

Exhibit A-1 HPOG 2.0 National Evaluation COVID-19 STS Research questions

Evaluation Component	Data Source1	Research Questions
Impact Evaluation	COVID-19 Cohort Short-term Follow-up Survey, Instrument 12a	There are two research question for the COVID-19 STS work. Are the impacts of HPOG 2.0 different for people who started training after the onset of the pandemic than for people trained before the outbreak?; and What are the post-COVID impacts on participants during this unique point in our economic history? This survey will also address the same overall research questions as the main study on the short-term impact of being offered an HPOG 2.0 slot. The questions of most relevance are: What is the impact on successful educational progress—defined as still enrolled in or having completed an education or training program? What is the impact on receipt of training; financial assistance for training, childcare, and financial assistance for childcare; and various forms of personal and supportive services such as tutoring, academic, or financial advising, or case management? What is the impact on receipt of credentials issued by the school/training program and on credentials or certifications issued by other organizations (e.g., a state licensing board)? What is the impact on participant employment, employment in a healthcare profession, hours of work, hours of work in a healthcare profession, receipt of employment benefits (e.g., health insurance, retirement, paid sick leave, paid vacation), and other terms of employment (e.g., shift work)? How does the impact on key outcomes—educational progress, productive activity, and earnings—vary with baseline (i.e., pre-randomization) characteristics of individuals, including gender, education, race/ethnicity, age, and receipt of public assistance?

The research questions are deliberately composite; that is, they ask about overall changes in levels and impacts from before COVID to during COVID. Those changes will likely be due to a combination of changes in program operation (e.g., program shutdowns, shifts to remote learning, limited availability of practicums), changes in the population applying, and changes in the labor market. Furthermore, those changes are likely inter-related (e.g., changes in program operation might induce changes in the population applying). To some extent, the analysis can understand the separate effect of different pathways. Thus, many characteristics of applicants are observed; both in the PAGES administrative data and in the survey. While those characteristics do not completely characterize differences in applicants, a shift-share analysis² will be performed and will likely be informative about the role of changes in applicants.

Study Design

The HPOG 2.0 National Evaluation is guided by the career pathways framework, as shown in the HPOG logic model (Attachment H). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012). These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports, and employer connections. The design for the HPOG 2.0 National Evaluation features a

• descriptive evaluation (including an implementation study, a systems study, and an outcome study);

• an impact evaluation (using a classic experimental design to measure and analyze key participant outcomes including completion of education and training, receipt of certificates and/or degrees, earnings, and employment in a healthcare career) (Instruments #5a and 5b approved in June 2017; Instrument #12 approved in July 2018; and Instrument 19 approved in July 2019 and Instruments 18 and 18a approved in June 2020); and

• a cost-benefit analysis (Instrument #20 approved in July 2019).

Exhibit A-2 provides a visual description of the major components and sub-components of the HPOG 2.0 National Evaluation. The short- and intermediate-term follow-up surveys are described in more detail in Supporting Statement A for the second and third revisions to OMB #0970-0462, approved in June 2018³ and July 2019⁴, respectively. The long-term follow-up survey instrument and supporting materials will be submitted for OMB review and approval under a forthcoming information collection request. The first part of this information collection request covers minor changes to the contact update procedures in support of the long-term follow-up study. The COVID-19 Cohort STS, the other component of this information request is described further below in Exhibit A-3.

Exhibit A-2: Components of the HPOG 2.0 National Evaluation

In August 2015, ACF received approval to collect a list of baseline data items from HPOG 2.0 National and Tribal Evaluation study participants. The continued use of that list was approved with each subsequent revision to OMB # 0970-0462. Subsequently OMB has approved:

• Contact update forms for survey respondents (Instruments #2-4, 5a and 5b approved in June 2017);

• The Short-term Follow-Up Survey and supporting materials (Instrument 12 approved in July 2018); and

• Descriptive evaluation protocols (Instruments 13-17 approved in July 2019); an Intermediate Follow-up Survey (Instruments 18 and 18a approved in June 2020); Phone-based Skills Assessment Pilot and Program Cost Survey (Instruments 19 and 20 also approved in July 2019).

This revised submission focuses only on the HPOG 2.0 National Evaluation⁵, for which data collection is ongoing, including the HPOG Participant Accomplishment and Grantee Evaluation System (PAGES) data collection, Participant Welcome Packets and Contact Update Forms, and the Intermediate Follow-up Survey.

Exhibit A-3 Overview of the HPOG 2.0 National Evaluation COVID-19 Cohort Short-term Survey

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Follow-up survey with HPOG 2.0 impact study participants enrolled after the onset of the COVID-19 pandemic	#12a COVID-19 Cohort Short-term Survey Supporting Materials	Respondents: 7,500 Participants, target 75 percent response rate Content: Training and employment history from randomization through date of interview School experiences Earned credentials Current/most recent job conditions, job quality, benefits, on-the-job training Income and economic well-being, student debt, financial resilience Adult well-being, including physical health, and housing conditions Household composition, family formation, and marital stability 21st Century skills/cognitive skills COVID-19 experiences Contact information Purpose: As with the Short-Term Follow-up Survey (Instrument #12), the COVID-19 Cohort STS will collect information on events after random assignment in many areas—particularly the receipt of training and related supports, and receipt of credentials. This survey information will be used as outcomes for the impact analysis. Of particular importance, this survey will assess the effect of the COVID-19 pandemic on HPOG 2.0 participants who enrolled after the onset of the pandemic relative to those who enrolled pre-pandemic. More broadly, job training is a common response to economic downturns—both by individuals and by policy makers. This study will increase understanding of the extent to which impacts vary with economic conditions. In addition, the pandemic itself might have encouraged some to seek healthcare training and discouraged others. Further, this survey will provide insight into the participant experiences with the program that will provide valuable information on program variations between those enrolled pre-pandemic and those enrolled after the onset of the pandemic.	Mode: Phone with in-person follow-up if feasible Duration: 60 minutes

Other Data Sources and Uses of Information

The research team will match participant data collected through the impact evaluation for both the treatment and control groups to employment and earnings data from the National Directory of New Hires (NDNH) and school enrollment and credential data from the National Student Clearinghouse (NSC).

These other data sources are complementary to the survey data. There is little explicit overlap. In particular, the administrative data provide high quality data for a limited set of outcomes.

NDNH data only has information on quarterly earnings (from which the evaluation infers employment) and receipt of UI benefits. A survey is needed to collect information on hours of work, hourly wage, benefits, working conditions, and sector of employment. The latter is crucial because increasing the healthcare workforce is an explicit statutory goal. At Q5, while the two systems provided similar estimated impacts on earnings (both near zero), the correlation in person-level quarterly earnings between the two systems at Q5 is just 0.49.

Finally, given the major goal of pre-COVID/COVID comparisons, it seems inappropriate to make changes in measurement. Such changes would make it hard to distinguish the effect of data source changes from the effect of true changes in the outcomes.

NSC data only include information on education in colleges (more precisely, degree granting institutions). Available evidence suggests that much of the early training in HPOG is at non-degree granting institutions. (As of the short-term survey, 14 percent of the treatment group had earned credentials from such training providers.) A survey is therefore crucial to capture a full sense of training received.

A3. Use of Information Technology to Reduce Burden

The HPOG 2.0 National and Tribal Evaluations have generated and will continue to generate a substantial amount of data using a combination of data collection methods. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. As described in the originally approved supporting statement (approved in August 2015, with revisions in January and July 2016, June 2017, and June 2018), participant- and grantee-level data are collected through PAGES, a cloud-based data system.

The HPOG 2.0 National Evaluation’s impact evaluation offers study participants the option to update their contact information online, by mail, or by telephone. The follow-up surveys are administered using computer assisted personal interviewing (CAPI) technology. CAPI technology allows interviewers to navigate through the survey faster, ensures that the skip patterns are properly implemented, and builds in checks to minimize data entry errors.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The evaluation team uses the quantitative data collected through PAGES, NDNH, and NSC to reduce the amount of data collected in the from participants during the follow-up survey.

A5. Impact on Small Businesses

There is no impact on small businesses.

A6. Consequences of Less Frequent Collection

ACF wants to understand the experiences of participants who apply for HPOG 2.0 after the onset of the COVID-19 pandemic. To estimate the program’s impact on other outcomes—the amount and type of training, employment outcomes such as the type of job, whether it is in healthcare, hours worked, benefits offered, and financial well-being—we need a one-time survey. In the absence of a survey, the evaluation team would only be able to estimate the program’s impact on outcomes for people enrolled after the onset of the pandemic as measured in NDNH and NSC administrative data.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 22, 2020, Volume 85, Number 256, page 83,586, and provided a sixty-day period for public comment. No public comments were received during the notice and comment period.

Consultation with Experts Outside of the Study

Please see previously approved revised submissions 1 and 3 for more detail on people consulted outside of the study team for previously approved information collection requests under OMB #0970-0462. This external feedback informed the previous studies as well as this information collection request. Abt Associates is actively exploring impacts of COVID on job training programs. In developing this submission, we consulted with two colleagues—Dr. Laura Peck and Dr. Dan Litwok—who are leading those efforts. Their comments (and their ongoing analyses) have informed this submission.

A9. Tokens of Appreciation

The HPOG 2.0 National Evaluation’s impact study is a longitudinal panel randomized controlled trial (RCT) study, intended to follow selected impact evaluation participants for at least fifteen months for those in the COVID-19 Cohort STS, and up to six years for those in the forthcoming Long-term Follow-up Study. Panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving sufficient sample size to maintain statistical power to detect meaningful effects in the analysis. Monetary tokens of appreciation show study participants that the researchers appreciate their continued involvement in the HPOG 2.0 National Evaluation information collection activities. The study team proposes to continue to use tokens of appreciation consistent with the design and amounts offered for other aspects of the impact study. (Please refer to the previously approved supporting statements for more details about previously approved tokens of appreciation for other HPOG 2.0 data collection activities and the associated justifications).

For the COVID-19 Cohort STS (Instrument #12a), the team hopes to achieve a response rate that is comparable to that achieved on the Short-term Follow-up Survey (Instrument #12). We made minimal changes to the instrument design to reduce the threat that differences in impacts observed for respondents enrolled after the onset of the pandemic, relative to that enrolled pre-pandemic, are caused by methodological differences rather than the pandemic. The same logic holds true with the use of tokens of appreciation for contact updates and survey completion.

Participants who respond to the contact update requests using Instrument 5b, will continue to receive a gift card valued at $5, via email, upon receipt of their updated information. None of the procedural changes requested in support of the Long-term Follow-up Study require changes in the previously approved token of appreciation structure.

The original Short-term Follow-up survey achieved a 74.4 percent completion rate, with a 5.1 percentage point differential between the treatment and control groups, and respondents received a $40 gift card. ACF proposes to use an incentive for the COVID-19 Cohort STS equal in value to that provided for the Short-term Survey—a gift card valued at $40, emailed to the respondent upon completion of the survey.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

The HPOG 2.0 National Evaluation collects the name, date of birth, social security number, address, phone number, and email information for all study participants through the PAGES system (previously approved under this OMB number). Name and contact information are necessary to aid in the contact update and survey data collection procedures. Social Security Numbers are used to aid in matching to the NDNH and NSC administrative databases.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier. For all analyses, study members are linked by one of two randomly generated IDs. One of these, the “AbtID” is used to link surveys across time, to the baseline administrative database, and to retrieved data from NSC. There is a “key” file that links every AbtID to the SSN. This key file is stored in a separate folder on Abt’s secure server. Only personnel with a strict need to know are allowed to access this folder. The other randomly generated ID is a pseudo-SSN created by the Office of Child Support Enforcement (OCSE) at ACF. They use it to allow Abt personnel to link survey data with NDNH information. Abt personnel do not have access to NDNH data with names or SSNs attached. Only OCSE personnel and their contractors process such files.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. All data collection protocols will receive IRB approval before data collection begins.

Data Security and Monitoring

As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor shall ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.

A11. Sensitive Information ⁶

For all the non-tribal grantees participating in the National Evaluation, study participants must provide an SSN in order to enroll in the program. The previously approved consent forms (Attachment B: National Evaluation Informed Consent Form A (Lottery Required)_REV, Attachment B: National Evaluation Informed Consent Form C (Lottery Required)_Verbal_REV, Attachment B2: Tribal Evaluation informed consent form A (SSNs), Attachment B2: Tribal Evaluation Informed Consent Form C (SSNs)_Verbal)) clearly state how SSNs will be used in the evaluation.⁷

Previously approved information collections for the HPOG 2.0 National Evaluation can be found in previously approved justification materials.

• PAGES Participant-Level Baseline Questions (Instrument 1) : see the original submission approved under OMB # 0970-0462.

• HPOG 2.0 Tribal Evaluation (Instruments 6-11): see the first revised submission approved under OMB # 0970-0462, approved in June 2017.

• HPOG 2.0 National Evaluation Descriptive Study (Instruments 2-4 and 13-17, plus Instrument 20 for the Cost-benefit study): see the submissions approved under OMB #0970-0462 in June 2017 and July 2019.

• HPOG 2.0 National Evaluation Impact Study (Previously approved Instruments 5a, 5b, 12, 18, 18a, and 19): see the submissions approved under OMB #0970-0462, approved in June 2018 and July 2019, respectively.

The COVID-19 Cohort STS (Instrument 12a) includes most of the same questions used in the original STS some of which may be seen as sensitive by some participants. Exhibit A-4 below provides an overview of the sensitive questions contained in the HPOG 2.0 National Evaluation impact evaluation’s COVID-19 Cohort STS.

All previously approved information collections were approved by the Abt Associates Institutional Review Board (IRB). The COVID-19 Cohort STS (Instrument 12a) and revised supporting materials Attachments K, M, and N, plus the new materials supporting the long-term follow-up study, Attachments AA and AB are in review with the Abt IRB.

Exhibit A-4 Sensitive Question Summary for COVID-19 Cohort STS

	Potentially Sensitive Questions and Topic Areas	Justification
HPOG 2.0 National Evaluation Impact Study COVID-19 Cohort STS (Instrument 12a)	Questions about income, employment status, receipt of government benefits, fertility, and household composition. Module on the effect the COVID-19 pandemic has had on participants.	As it is hoped that HPOG 2.0 will have favorable impacts in all these areas—and it is recognized that the pandemic could affect the outcomes for those who enrolled after the onset of the pandemic relative to those enrolled prior to the pandemic—failure to ask any of these questions would limit the findings of the evaluation. Questions about respondents’ personal experiences during the pandemic could be viewed as sensitive. Interviewers will remind study members during the interview that they may refuse to answer individual items. Interviewers will also remind study members that their responses will be kept private to encourage their candid responses. Interviewers are also trained to recognize when questions are upsetting to participants and how to deescalate the situation swiftly.

A12. Burden

Explanation of Burden Estimates

The burden estimates presented here first update the remaining burden on previously approved instruments still in use, followed by estimated burden for new instruments.

Burden Remaining from Previously Approved Instruments

At the time of this request (April 2021), only instrument #1 (PAGES baseline and program operations data), Instruments 5a and 5b (Welcome to the Study Packet and Participant Contact Update Form—supporting both the COVID-19 Cohort STS and the forthcoming Long-term Survey), and Instruments 18 and 18a (Intermediate Follow-up Survey) are still active. The remaining burden estimates for those instruments are summarized below in Exhibit A-5:

Exhibit A-5 Remaining Burden on Previously Approved Instrument

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instrument 1: PAGES Grantee- and Participant-Level Data Collection (all grantees)	32	2	31.75	2,032	677	$28.29	$19,152.33
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at non-tribal grantees)	4,860	1	.5	2,430	810	$3.94	$3,191.40
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at Tribal grantees)	430	1	.25	108	36	$3.94	$141.84
Instrument 5a: HPOG 2.0 National Evaluation Welcome to the Study Packet	7,500	1	.1	750	250	$10.15	$2,537.50
Instrument 5b: HPOG 2.0 National Evaluation Letter and Participant Contact Update Form/Instrument 5b: HPOG 2.0 National Evaluation Contact Update Form Phone Version	2,244	7	.1	1,571	524	$10.15	$5,318.60
Instrument 18: Intermediate Follow-up Survey for the National Evaluation impact study	1,138	1	.92	1,047	349	$10.15	$3,542.35
Instrument 18a: HPOG 2.0 National Evaluation Intermediate Follow-up Survey – Critical Items Only	125	1	.33	41	14	$10.15	$142.10
Total	16,329			7,979	2,650		$34,026.12

Burden Estimates for This Information Collection Request

Instrument #12a, COVID-19 Cohort STS, the subject of this information collection request, is virtually identical in design and purpose to previously approved instrument #12, the Short-term Follow-up Survey. Instrument #12 took 60 minutes to complete; thus, the average burden per response for Instrument #12a will also average 60 minutes in length. Exhibit A-6 summarizes the new burden for this information collection request. The total number of respondents is based on total sample of 7,500 and a projected response rate of 75 percent.

Exhibit A-6 Burden Estimate for New Information Collection Request

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Avg. Hourly Wage Rate	Total Annual Respondent Cost
Instrument 12a: COVID-19 Cohort STS	5,625	1	1	5,625	1,875	$10.15	$19,031.25
Total	5,625	1	1	5,625	1,875		$19,031.25

Estimated Annualized Cost to Respondents

We used the same wage rates for calculating annualized cost to respondents that we have used in all previously approved revisions to OMB# 0970-0462. Please refer to previously approved versions of Supporting Statement A for more detail on how the wage rate calculations were done. Based on these wage rates, the evaluation team estimates that the annual costs for the remaining previously approved data collection and this new information collection request is $53,057.37 annualized over the three-year request for clearance. The total remaining costs are $159,172.11.

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The costs in Exhibit A-7 below are estimates to proceed with the COVID-19 Cohort STS based on the evaluation team’s experiences conducting the prior short-term and intermediate follow-up survey data collection efforts conducted under OMB # 0970-0462. The exhibit also includes the remaining costs to complete the previously approved information collections (PAGES intake and program monitoring, ongoing contact updates, and the intermediate follow-up survey).

Exhibit A-7 Preliminary Cost Estimates for this Information Collection Request

Activity	Detail	Estimated Cost
Survey Design Instrument Development Pilot and User Testing OMB Clearance	FTE time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses which would not have been incurred without this collection of information	$245,500
Survey administration and Participant Contact Update Activities	FTE time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses which would not have been incurred without this collection of information	$6,754,450
Analysis and initial dissemination	FTE time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses which would not have been incurred without this collection of information	$1,747,000
Total costs over the request period		$8,746,950
Annual costs		$2,915,650

A15. Reasons for changes in burden

This request does not include any changes in burden to previously approved instruments. Exhibit A-5 shows the remaining burden for the previously approved information collections that are still active (Instrument 1, Instrument 5a and Instrument 5b, and Instruments 18 and 18a)⁸. As shown in Exhibit A-6, this request for clearance is for a new information collection under OMB #0970-0462, the COVID-19 STS (Instrument #12a). The respondents for the COVID-19 STS were not included in the survey cohorts included in the previously approved burden estimates.

A16. Timeline

ACF awarded grants to 32 grantee organizations in 2015. Participant enrollment began in February 2016 and will continue through September 2021. OMB approved several information collections requests under OMB #0970-0462 in support of the HPOG 2.0 National and Tribal Evaluations. The timeline for OMB approval, data collection, reporting, and archiving for each component of the HPOG 2.0 National and Tribal Evaluations are summarized by study component in Exhibit A8 below.

Exhibit A-8: Project Schedule for Data Collection, Analysis, Publication, and Archiving⁹

Task	Timing	Data to be Archived (yes/no)
HPOG 2.0 National Evaluation: Descriptive Evaluation (Instrument #s 2-4 and 13-17 approved in June 2017 and July 2019)
Data Collection
Descriptive evaluation data collection for implementation and systems studies, and for in-depth participant interviews	June 2017-January 2019	Grantee and Participant Interview Data will be archived; Systems Interview Data will not be archived because it would be difficult to anonymize given small number of sites, detailed information on partners, and small number of respondents.
National evaluation descriptive study site visits	Fall 2018	No, the data would be difficult to anonymize due to small number of sites and respondents.
Reporting
Systems Study Report	Late 2021
Implementation Study Report	Late 2021
Participant Interview Briefs	Ongoing beginning Summer 2021
Outcomes Report	Late 2022
HPOG 2.0 National Evaluation: Impact Evaluation (Instrument 5a and 5b, 12, 18 and 18a (approved in June 2017, June 2018, and July 2019)
Data Collection
Contact update mailing for survey sample (Instrument 5a and 5b)	Quarterly beginning 3 months after random assignment up to 3 months before the survey fielding period (November 2017-Winter 2023)	No, this only collects participant PII, there is no way to anonymize it.
Short-term (15-Month) Follow-up Survey Data Collection	October 2018-November 2019 (15 months after randomization for participants enrolled March 2017-February 2018)	Yes, after publication of the Short-term Impact Report
Intermediate (36-Month) Follow-up Survey Data Collection (Instrument 18 and 18a)	Ongoing—began in September 2020 will conclude in June 2021	Yes, after publication of the Intermediate Impact Report
COVID-19 Cohort Short-term (15-Month) Follow-up Survey Data Collection (Instrument 12a)	Planned November 2021-March 2023	Yes, after publication of the COVID-19 Cohort Impact Report
Long-term Follow-up Survey (forthcoming)	Planned May-October 2023	Yes, after publication of the Long-term Impact Report
Reporting
Draft Short-term Impact Report to ACF	Summer 2020
Draft Intermediate Impact Report to ACF	Spring 2022
Draft COVID-19 Cohort Impact Report	Spring 2024
Draft Longer-term Impact Report to ACF	Fall 2024
HPOG 2.0 National Evaluation: Cost-benefit analysis study (Instrument #20, Approved in July 2019)
Program cost data collection	Fall 2019 – Spring 2020	No
Draft Cost-Benefit report to ACF	Spring 2024
HPOG 2.0 National and Tribal Evaluation Participant Accomplishment and Grantee Evaluation System (PAGES) (Instrument #1, Approved in August 2015)
PAGES grantee-level and ongoing participant-level data collection	September 2015 – Winter 2022	Yes
Reporting
6 Semi-annual Performance Progress Reports	September 2015 – September 2021
Five annual reports	September 2015 – September 2020

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

New Instrument:

• Instrument 5b: HPOG 2.0 Contact Update Form Phone Version

• Instrument 12a: COVID-19 Cohort Short-term Follow-up Survey

Previously Approved Instruments Still in Use:

• Instrument 1: PAGES Grantee- and Participant-Level Data Items List

• Instrument 5: HPOG 2.0 National Evaluation welcome packet and participant contact update forms

• Instrument 5a: HPOG 2.0 National Evaluation welcome packet and contact update form_REV Instrument 5b: HPOG 2.0 National Evaluation participant contact update letter and form

• Instrument 18: HPOG 2.0 Intermediate Follow-up Survey_ REV_June2020

  • Instrument 18a: HPOG 2.0 Intermediate Follow-up Survey_Critical Items Only

Previously Approved Instruments No Longer in Use:

• Instrument 2: HPOG 2.0 National Evaluation Screening Interview

• Instrument 3: HPOG 2.0 National Evaluation first-round telephone interview protocol

• Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews

  • Instrument 4A HPOG 2.0 National Evaluation In-Person Implementation Interview

  • Instrument 4B HPOG 2.0 National Evaluation In-Person Implementation Interviews Basic Skills Training

  • Instrument 4C HPOG 2.0 National Evaluation In-Person Implementation Interviews Career Pathways

  • Instrument 4D HPOG 2.0 National Evaluation In-Person Implementation Interviews Work-Readiness

  • Instrument 4E HPOG 2.0 National Evaluation In-Person Implementation Interviews Sustainability

• Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews

• Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews

• Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews

• Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups

• Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews

• Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews

• Instrument 12: HPOG 2.0 National Evaluation Short-term Follow-up Survey

• Instrument 13: HPOG 2.0 Screening Interview Second Round

• Instrument 14: HPOG 2.0 Second Round Telephone Interview Guide

• Instrument 15: HPOG 2.0 Program Operator Interview Guide for Systems Study

• Instrument 16: HPOG 2.0 Partner Interview Guide for Systems Study

• Instrument 17: HPOG 2.0 Participant In-depth Interview Guide

• Instrument 19: HPOG 2.0 Phone-based Skills Assessment Pilot Study Instrument

• Instrument 20: HPOG 2.0 Program Cost Survey

New Attachments

• Attachment K-Revised: COVID-19 Cohort Short-term Survey Advance Letter

• Attachment L-Revised: COVID-19 Short-Term Survey Sources

• Attachment M-Revised: COVID-19 Cohort Short-term Survey Trying to Reach You Flyer

• Attachment N-Revised: COVID-19 Cohort Short-Term Survey Email Reminder Text

• Attachment AA: HPOG 2.0 Participant Newsletter

Previously Approved Attachments Still in Use

• Attachment A: References

• Attachment B: New Informed Consent Forms, Updated Time Period

  • Attachment B: National Evaluation Informed Consent Form A (Lottery Required) _REV

  • Attachment B: National Evaluation Informed Consent Form C (Lottery Required) _Verbal_REV

  • Attachment B2: Tribal Evaluation informed consent form A (SSNs)

  • Attachment B3: Tribal Evaluation informed consent form B (Unique identifiers)

  • Attachment B2: Tribal Evaluation Informed Consent Form C (SSNs)_Verbal

  • Attachment B3: Tribal Evaluation Informed Consent Form D (Unique identifiers) _Verbal

• Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter_REV

• Attachment Q: Intermediated Follow-up Survey Sources_REV

• Attachment R: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Trying to Reach You Flyer

• Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder_REV

Previously Approved Attachments No Longer in Use

• Attachment B: Previously Approved Informed Consent Forms

  • Attachment B: National Evaluation informed consent form A (Lottery Required)

  • Attachment B: National Evaluation informed consent form B (Lottery Not Required)

  • Attachment B: National Evaluation Informed Consent Form C (Lottery Required) _Verbal

  • Attachment B: National Evaluation Informed Consent Form D (Lottery Not Required) _Verbal

• Attachment C: 60-Day Federal Register Notice

• Attachment D: Previously Approved Sources and Justification for PAGES Grantee- and Participant-Level Data Items

• Attachment E: Previously Approved Final Updated Attachment E PPR Data List and Mockup

• Attachment F: First Round of HPOG Grantees Research Portfolio

• Attachment G: Previously Approved Participant Contact Information Update Letter and Form (Obsolete, replaced by Instrument 5a and 5b)

• Attachment H: HPOG Logic Model

• Attachment I: Previously Approved Focus Group Participant Consent Form

• Attachment I: New Focus Group Participant Consent Form_Remote

• Attachment J: Previously Approved Interview Verbal Informed Consent Form

• Attachment J: New Interview Verbal Informed Consent Form_Remote

• Attachment K: HPOG 2.0 National Evaluation Short-term Follow-up Survey Advance Letter

• Attachment L: HPOG 2.0 National Evaluation Short-term Follow-up Survey Sources

• Attachment M: HPOG 2.0 National Evaluation Short-term Follow-up Survey Trying to Reach You Flyer

• Attachment N: HPOG 2.0 National Evaluation Short-term Follow-up Survey Email Reminder

• Attachment O: Research Questions for Previously Approved Data Collection Efforts (National Evaluation and Tribal Evaluation)

• Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter

• Attachment Q: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Sources

• Attachment R: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Trying to Reach You Flyer

• Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder

1 Additional data sources will be used to answer the research questions, beyond those listed here. Only the data sources relevant to this submission are listed.

2 In this case, a shift-share analysis would explore how much of the change in levels and in impacts can be explained by composition. Specifically, the data would be reweighted so that the COVID sample has the same characteristics as the pre-COVID sample (or vice versa).

3 ICR Ref. No: 201802-0970-001 

4 ICR Ref. No: 201904-0970-006 

5 Please refer to OMB #0970-0462, Revised submission #1, approved in June 2017 for more detail on the HPOG 2.0 Tribal Evaluation, for which data collection is complete.

6 Examples of sensitive topics include (but are not limited to): Social Security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships ( e.g., family, pupil-teacher, employee-supervisor); mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment, WIC. or SNAP); and immigration/citizenship status.

7 Attachment B3: Tribal Evaluation informed consent form B (Unique identifiers) and Attachment B3: Tribal Evaluation Informed Consent Form D (Unique identifiers) _Verbal are used by tribal grantees that do not collect SSNs.

8 All data collection for instruments 2-4, 6-17, and 19-20 is now complete. Since they are complete, they are no longer reflected in Exhibit A-5 as they have no remaining burden.

9 Tribal Evaluation data collection using Instruments 6-11, approved in June 2017, and renewed in subsequent revisions under this OMB # took place between Summer 2017 and Fall 2020. Reports were published annually after each data collection. Final report preparation is underway. Archiving of the tribal data is still under discussion.

17