HPOG 2.0_Supporting Statement A_Revised_NonSubChange November 2020_Tracked to ACF_Revised_clean
HPOG 2.0_Supporting Statement A_Revised_NonSubChange November 2020_Tracked to ACF_Revised_clean.docx
OPRE Evaluation - National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants [descriptive evaluation, impact evaluation, cost-benefit analysis study, pilot study]
OMB: 0970-0462
Supporting Statement for OMB Clearance Request
Part A
National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants
0970-0462
Revised April 2019
Revised July 2019
Revised April 2020
Revised June 2020
Revised November 2020
Submitted by:
Office of
Planning,
Research & Evaluation
Administration for Children & Families
U.S. Department of
Health
and Human Services
Federal Project Officers:
Hilary Bruck
Nicole Constance
Amelia Popham
Table of Contents
A1: Necessity for the Data Collection 6
A2: Purpose of Survey and Data Collection Procedures 8
A3: Improved Information Technology to Reduce Burden 21
A4: Efforts to Identify Duplication 22
A5: Involvement of Small Organizations 23
A6: Consequences of Less Frequent Data Collection 24
A8: Federal Register Notice and Consultation 24
A9: Incentives for Respondents 25
A10: Privacy of Respondents 31
A12: Estimation of Information Collection Burden 35
A13: Cost Burden to Respondents or Record Keepers 38
A14: Estimate of Cost to the Federal Government 38
A16: Plan and Time Schedule for Information Collection, Tabulation and Publication 40
A17: Reasons Not to Display OMB Expiration Date 44
A18: Exceptions to Certification for Paperwork Reduction Act Submissions 44
Instruments
Revised Instruments Included in this Request
Instrument 5a: HPOG 2.0 National Evaluation welcome packet and contact update form_REV
Previously Approved Instruments
Instrument 1: PAGES Grantee- and Participant-Level Data Items List
Instrument 2: HPOG 2.0 National Evaluation Screening Interview
Instrument 3: HPOG 2.0 National Evaluation first-round telephone interview protocol
Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews
Instrument 4A HPOG 2.0 National Evaluation In-Person Implementation Interview
Instrument 4B HPOG 2.0 National Evaluation In-Person Implementation Interviews Basic Skills Training
Instrument 4C HPOG 2.0 National Evaluation In-Person Implementation Interviews Career Pathways
Instrument 4D HPOG 2.0 National Evaluation In-Person Implementation Interviews Work-Readiness
Instrument 4E HPOG 2.0 National Evaluation In-Person Implementation Interviews Sustainability
Instrument 5: HPOG 2.0 National Evaluation welcome packet and participant contact update forms
Instrument 5b: HPOG 2.0 National Evaluation participant contact update letter and form
Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews
Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews
Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews
Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups
Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews
Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews
Instrument 12: HPOG 2.0 National Evaluation Short-term Follow-up Survey
Instrument 13: HPOG 2.0 Screening Interview Second Round
Instrument 14: HPOG 2.0 Second Round Telephone Interview Guide
Instrument 15: HPOG 2.0 Program Operator Interview Guide for Systems Study
Instrument 16: HPOG 2.0 Partner Interview Guide for Systems Study
Instrument 17: HPOG 2.0 Participant In-depth Interview Guide
Instrument 18: HPOG 2.0 Intermediate Follow-up Survey_ REV_June2020
Instrument 18a: HPOG 2.0 Intermediate Follow-up Survey_Critical Items Only
Instrument 19: HPOG 2.0 Phone-based Skills Assessment Pilot Study Instrument
Instrument 20: HPOG 2.0 Program Cost Survey
Attachments
Previously Approved Attachments
Attachment A: References
Attachment B: Previously Approved Informed Consent Forms
Attachment B: National Evaluation informed consent form A (Lottery Required)
Attachment B: National Evaluation informed consent form B (Lottery Not Required)
Attachment B: National Evaluation Informed Consent form C (Lottery Required)_Verbal
Attachment B: National Evaluation Informed Consent form D (Lottery Not Required)_Verbal
Attachment B: New Informed Consent Forms, Updated Time Period
Attachment B: National Evaluation Informed Consent Form A (Lottery Required)_REV
Attachment B: National Evaluation Informed Consent Form C (Lottery Required)_Verbal_REV
Attachment B2: Tribal Evaluation informed consent form A (SSNs)
Attachment B3: Tribal Evaluation informed consent form B (Unique identifiers)
Attachment B2: Tribal Evaluation Informed Consent form C (SSNs)_Verbal
Attachment B3: Tribal Evaluation Informed Consent form D (Unique identifiers)_Verbal
Attachment C: 60-Day Federal Register Notice
Attachment D: Previously Approved Sources and Justification for PAGES Grantee- and Participant-Level Data Items
Attachment E: Previously Approved Final Updated Attachment E PPR Data List and Mockup
Attachment F: First Round of HPOG Grantees Research Portfolio
Attachment G: Previously Approved Participant Contact Information Update Letter and Form (Obsolete, replaced by Instrument 5a and 5b)
Attachment H: HPOG Logic Model
Attachment I: Previously Approved Focus Group Participant Consent Form
Attachment I: New Focus Group Participant Consent Form_Remote
Attachment J: Previously Approved Interview Verbal Informed Consent Form
Attachment J: New Interview Verbal Informed Consent Form_Remote
Attachment K: HPOG 2.0 National Evaluation Short-term Follow-up Survey Advance Letter
Attachment L: HPOG 2.0 National Evaluation Short-term Follow-up Survey Sources
Attachment M: HPOG 2.0 National Evaluation Short-term Follow-up Survey Trying to Reach You Flyer
Attachment N: HPOG 2.0 National Evaluation Short-term Follow-up Survey Email Reminder
Attachment O: Research Questions for Previously Approved Data Collection Efforts (National Evaluation and Tribal Evaluation)
Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter
Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter_REV
Attachment Q: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Sources
Attachment Q: Intermediated Follow-up Survey Sources_REV
Attachment R: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Trying to Reach You Flyer
Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder
Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder_REV
Attachment T: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot flyer
Attachment U: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot grantee letter
Attachment V: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot participant letter
Attachment W: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot recruitment script
Attachment X: Complete list of previously approved data collection instruments
Attachment Y: 60-day Federal Register Notice
Attachment Z: Participant Interview Recruitment Materials
Overview The
revisions in this Supporting Statement reflect changes requested as
part of a non-substantive change request, submitted to OMB in
November 2020.
Over
the life of the HPOG 2.0 National and Tribal Evaluations OMB has
reviewed and approved 20 unique data collection instruments and
associated supporting materials under OMB Control Number 0970-0462.
As
described in an accompanying memo, ACF is seeking approval for
non-substantive changes to previously approved items under the
National Evaluation. ACF is considering funding a new survey cohort
of participants enrolled in HPOG 2.0 National Evaluation after the
onset of the COVID-19 pandemic and through the end of the HPOG 2.0
grant period, and these changes are needed to support that effort
if it is funded. If funded, there would be a public comment period
associated with the instruments and supporting materials related to
that information collection request. The non-substantive changes
included in this submission are: Minor
revisions to the National
Evaluation Welcome to the Study Packet and Contact Update Form
(Instrument 5a) and an increase in number of respondents; and Increase
number of respondents for Instrument 5b: Participant Contact
Update Letter and Form.
Part A: Justification
The National and Tribal Evaluations of the Health Profession Opportunity Grants (HPOG) program, known as the HPOG 2.0 Evaluation is funded by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS).
The HPOG 2.0 Evaluation includes evaluations of tribal grantees (Tribal Evaluation) and non-tribal grantees (National Evaluation). Since the initial approval in August 2015, OMB has approved a wide array of information collection tools (20 different instruments and associated supporting materials) and procedures in support of the HPOG 2.0 National and Tribal Evaluations under OMB Control N0.0970-0462. Exhibit A-1 summarizes prior requests to OMB for approval of new instruments and the most recent non-substantive change request. The exhibit then summarizes the changes to several previously approved instruments, attachments, and procedures reflected in this request. Justification for these non-substantive changes can be found in the supplementary document OMB#0970-0462_NonSubstantiveChange Request Memo_November2020.docx.
Exhibit A-1: Clearance Requests and Instruments for HPOG 2.0 (OMB Control No. 0970-0462)
Request |
Instrument(s) |
Request Date |
Approval Date |
Link to Supporting Statement |
Original |
Participant Accomplishment and Grant Evaluation System (PAGES) (Instrument #1) |
5/13/15 |
8/6/15 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201505-0970-002 |
1st Rev. |
Various baseline, process and contact update forms (Instruments #2-5b for the National Evaluation; #6-11 for the Tribal Evaluation) |
10/26/16 |
6/27/17 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201610-0970-012 |
2nd Rev. |
National Evaluation Short-term Follow-Up Survey (Instrument #12) |
2/5/18 |
6/8/18 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201802-0970-001 |
3rd Rev. |
Additional instruments for the National Evaluation: Descriptive evaluation protocols (Instruments #13-17); Intermediate Follow-up Survey (Instrument #18); Phone-based Skills Assessment Pilot (Instrument #19); and Program Cost Survey (Instrument #20). |
4/23/2019 |
7/24/19 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201904-0970-006
|
Non-substantive Change Request (this submission) |
Non-substantive change request for minor changes and modest increase in the number of respondents to two previously approved instruments in support of the HPOG 2.0 National Evaluation:
|
November 2020 |
TBD |
|
A1: Necessity for the Data Collection
In August 2015, ACF received approval to collect baseline data from HPOG 2.0 National and Tribal Evaluation study participants using Instrument #1 PAGES Grantee and Participant-level Data Items List. The continued use of that list was approved with each subsequent revision to OMB Control Number 0970-0462. In June 2017, ACF received approval for data collection Instruments 5a and 5b, and procedures. The continued use of Instrument 5b was approved with each subsequent revisions to OMB Control Number 0870-0462. In July 2019 ACF received approval for several data collection activities to support the HPOG 2.0 National Evaluation.
This submission requests approval of non-substantive changes to increase the number of respondents for two previously approved instruments and minor revisions to one of them. Specifically this request covers:
National Evaluation
Minor revisions to the National Evaluation Welcome to the Study Packet and Contact Update Form (Instrument 5a);
A modest increase in the number of respondents for the previously approved Instrument 5a Welcome to the Study Packet and Contact Update Form, to support a new survey cohort of participants enrolled in HPOG 2.0 National Evaluation during COVID-19 pandemic; and
A modest increase in the number of respondents for the previously approved Instrument 5b Participant Contact Update Letter and Form (recently renewed in the July 2019 approval) to support the collection of updated contact information for the new survey cohort.
A1.1 Study Background
The HPOG Program, established by the Patient Protection and Affordable Care Act of 2010 (ACA), funds training in high-demand healthcare professions, targeted to TANF recipients and other low-income individuals. The HPOG Program is administered by the ACF Office of Family Assistance (OFA). The first round of HPOG grants was awarded in 2010. In September 2015, OFA awarded a second round of HPOG grants—valued at approximately $72 million—to 32 organizations located across 21 states. Grantees include six community based organizations, four state government entities, seven local workforce development agencies, ten institutions of higher education, two tribal colleges, one tribal human service agency, one tribe, and one Indian Health Board. Those 32 grantees oversee 43 individual HPOG programs—five tribal programs and 38 non-tribal programs. All grantees are participating in this federal evaluation.
HPOG programs: (1) target skills and competencies demanded by the healthcare industry; (2) support career pathways, such as an articulated career ladder; (3) result in an employer- or industry-recognized credential (which can include a license, third-party certification, postsecondary educational certificate or degree, as well as a Registered Apprenticeship certificate); and (4) combine supportive services with education and training services to help participants overcome barriers to employment, as necessary.
HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. See A1.2 for additional information. The federal evaluation activities are intended to expand the career pathways evidence base and to build on what has been learned to date about how to design and implement successful career pathways programs for low-income and low-skilled individuals, and improve the outcomes of individuals who participate in these programs. The federal evaluation for the non-tribal HPOG 2.0 grantees involves random assignment of individual participants. Tribal grantees are participating in a coordinated evaluation that does not involve random assignment.
Abt Associates is the prime contractor and the lead for the HPOG 2.0 National Evaluation. Abt and the Urban Institute led the design of the Participant Accomplishment and Grant Evaluation System (PAGES), which collects and stores uniform data needed for performance management and the federal evaluations, incorporating the required semi-annual grantee performance reports to ACF (Attachment E). Both organizations are overseeing PAGES data collection. Partners MEF Associates, Insight Policy Research and the Urban Institute are assisting with the site monitoring, descriptive evaluation, and cost-benefit analysis. NORC at the University of Chicago is leading the HPOG 2.0 Tribal Evaluation under subcontract to Abt Associates.
A1.2 Legal or Administrative Requirements that Necessitate the Collection
The HPOG Program is authorized under section 2008(a)(1) of the Social Security Act (42 U.S.C. § 1397g(a)(1)), and extended by the “Coronavirus Aid, Relief, and Economic Security Act” or the “CARES Act of 2020, Pub. L. 116-136 through the first two months of fiscal year 2021. H.R. 3590, the ACA requires an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(C)). The Act calls for evaluation activities to assess the success of HPOG in “creating opportunities for developing and sustaining, particularly with respect to low-income individuals and other entry-level workers, a health professions workforce that has accessible entry points, that meets high standards for education, training, certification, and professional development, and that provides increased wages and affordable benefits, including healthcare coverage, that are responsive to the workforce’s needs” (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)).
There were two Funding Opportunity Announcements (FOAs) for the second round of HPOG grants—one for non-tribal grantees (HHS-2015-ACF-OFA-FX-0951) and one for Tribal grantees (HHS-2015-ACF-OFA-FY-0952). Both FOAs required all HPOG 2.0 grantees to participate in a federal evaluation and to follow all evaluation protocols established by ACF or its designated contractors. Participating in the federal evaluations includes the use of the PAGES data system to collect uniform data elements and, for non-tribal grantees, the facilitation of random assignment.
HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. Accordingly, ACF is rigorously evaluating the effectiveness of funded HPOG 2.0 programs. Data collected under PAGES is used to: automatically generate the federally required semi-annual program performance reports; inform ACF reports to Congress; monitor and manage the performance of the grant-funded projects; inform the HPOG 2.0 National Evaluation impact, outcomes, and implementation studies and HPOG 2.0 Tribal Evaluation; and inform other future research and evaluation efforts. Data collected through the previously approved instruments support the HPOG 2.0 National and Tribal Evaluations.
A2: Purpose of Survey and Data Collection Procedures
A2.1 Overview of Purpose and Approach
The data collection instruments described in this section were previously approved in July 2019, with the verbal consent form revisions approved in April 2020 and other modifications approved in June 2020. This submission seeks approval of non-substantive changes to the previously approved Instrument 5a, Welcome to the Study Packet and Contact Update Form, and minor increases in the number of respondents for Instrument 5a and Instrument 5b Participant Contact Update Letter and Form. Details on these changes can be found in this section and throughout this Supporting Statement A, in Supporting Statement B, Sections B2, B3 and B4 and the supporting memo OMB#0970-0462_NonSubstantiveChangeRequest Memo_November2020.docx.
HPOG 2.0 Informed Consent-National and Tribal Evaluations
The informed consent forms were originally approved in August 2015, and subsequently revised most recently in April 2020. (Please see Supporting Statements A and B approved in April 2020 for more detail.) OMB approved minor wording changes in two of the National Evaluation Informed Consent Forms—Form A (written consent for those subject to the lottery) and Form C (verbal consent for those subject to the lottery) in June 2020. ACF is planning a longer data collection period for the National Evaluation than originally anticipated. This change prompted ACF to seek approval to revise the following paragraph regarding the types of administrative data the evaluation will obtain and how participant data will be used in both Informed Consent Form A and C for the National Evaluation:
“Information from government sources so researchers can learn more about your future employment, earnings, and post-secondary education over the next few years. Abt will use your name and social security number to get some of these data from the National Directory of New Hires and the National Student Clearinghouse. We will collect these data for you and up to 52,000 other study participants. . The researchers will collect data covering the period before you enrolled in the study and continuing through the end of HHS’s research on HPOG.”
Study participants that already signed consent forms with the five year restriction will receive a letter noting that ACF’s plans have evolved over time to include a longer data collection period. Revised versions of the consent forms can be found in Attachment B, National Evaluation Informed Consent Form A (Lottery Required) and National Evaluation Informed Consent form C (Lottery Required)_Verbal.
HPOG 2.0 National Evaluation
The National Evaluation involves random assignment of individual participants. As stated in the FOA, the non-tribal HPOG 2.0 grantees are required to abide by random assignment procedures and facilitate the random assignment process for individuals by entering eligible HPOG program applicants into a lottery to determine if they will be invited to participate in the program.
Applicants who are not invited to participate serve as a control group in the evaluation. The control group members do not receive HPOG program services, but may enroll in any other program or service for which they are eligible. Individuals must complete the application process prior to random assignment; only individuals who have been deemed eligible for program participation may be entered into the lottery.
For the National Evaluation, the third revised ICR submission, approved in July 2019, covered instruments for the descriptive evaluation, the impact evaluation, and the cost-benefit analysis study. The eight instruments in the third revised submission, approved in July 2019, were:
Five instruments for the descriptive evaluation (Instruments 13-17)
Screening Interview Guide, Second-round Interviews,
Second-round Telephone Interview Guide,
Program Operator Interview Guide for systems study,
Partner Interview Guide for systems study, and
Participant In-depth Interview Guide;
Two instruments for the impact evaluation (Instruments 18 and 19)
Intermediate Follow-up Survey,
Phone-based Skills Assessment Pilot; and
One instrument for the cost-benefit analysis study (Instrument 20)
Program cost survey.
Data collection for all of the instruments previously approved in July 2019 is now complete, with the exception of Instrument 18, the Intermediate Follow-up Survey. OMB also approved the modifications to Instrument 18 listed below in June 2020:
Selecting the Final Assessment Questions in Section J. The evaluation team completed the pilot assessment data collection and analysis in January 2020. ACF proposes to retain 11 verbal and 11 math skill questions, with varying degree of difficulty. ACF seeks approval to drop the remaining items conditionally approved in the Intermediate Follow-up Survey: J10, J13-J16, J20, J22, J25-J28, J31, J35-J36, J40-J41, and J43-J44.
Additional Questions to Improve Data Quality. Two items were unintentionally omitted from the survey but are important to the Cost-Benefit Analysis. ACF also proposes to add a small number of questions to Instrument 18 to collect data on the effects of COVID-19 on study participants’ education, training, and employment patterns since the beginning of the outbreak in early 2020.
Cuts to Reduce Administration Time. The original burden estimate for the Intermediate Follow-up Survey, as approved in July 2019, was 60 minutes, but revised estimates based on experience administering the Short-term Follow-up Survey, which includes many of the same items, suggest the Intermediate Follow-up Survey interview length will be closer to 67 minutes. ACF proposes to drop several questions in order to reduce the overall length.
New Version of Instrument 18 Intermediate Follow-up Survey (Instrument 18a-Critical Items Only.) ACF requests approval for a second version of Instrument 18 that is significantly shorter and captures only the most critical outcomes. This version of Instrument 18 can be completed in just 20 minutes, and will be used with individuals who refuse to respond to the full survey. The shorter version of Instrument 18 will be offered as the last refusal conversion effort in an effort to boost response rates. Supporting Statement B, Section B2 provides more detail on the administration of Instrument 18a.
OMB also approved the request for revisions to two of the supporting materials for the Intermediate Follow-up Survey, Attachment P the Intermediate Follow-up Survey Advance Letter_REV and Attachment S HPOG 2 Intermediate Survey Email Reminder Text_REV in June 2020.
This non-substantive change request seeks approval to increase the number of respondents for the previously approved Welcome to the Study Packet and Contact Update Form (Instrument 5a) and Participant Contact Update Letter and Form (Instrument 5b), and some minor wording changes to Instrument 5a. These changes are necessary because ACF is considering fielding a survey to a new cohort of study participants—those enrolled after the onset of the COVID-19 Pandemic. If funded, this cohort will allow ACF and the evaluation contractor to better understand the effects of COVID-19 on HPOG 2.0 program and participant impacts.
HPOG 2.0 Tribal Evaluation
There are no changes to the Tribal Evaluation included in this non-substantive change request. The changes summarized below were approved in June 2020.
In July 2016, OMB approved the HPOG 2.0 Tribal Evaluation informed consent forms (Attachments B2 and B3). In June 2017, OMB approved the HPOG 2.0 Tribal Evaluation data collection instruments (Instruments 6-11) and consent forms for the focus groups (Attachment I) and interviews (Attachment J). Under the previously approved Tribal Evaluation study procedures, qualitative interviews and focus groups occur in person at each of the five Tribal HPOG 2.0 grantee sites. The fourth and final round of site visits was planned for spring 2020. It is not likely that the remaining site visits can occur in-person as a result of social distancing guidelines and travel restrictions implemented nationwide to stop the spread of COVID-19. However, Tribal HPOG 2.0 grantee staff and partners are continuing to implement their programs remotely, and it is important that the tribal evaluation team capture for the HPOG 2.0 Tribal Evaluation information about this period of program implementation. In the event that in person data collection in summer 2020 is not feasible, ACF requests approval to conduct remote data collection by phone or virtual meeting in addition to in person.
OMB approved additional versions of the previously approved focus group and interview consent forms (see Attachment I_Focus Group Informed Consent Form_Remote and Attachment J Interview Verbal Informed Consent Form_Remote) for use during remote data collection in June 2020. The remote version of the informed consent form in Attachment I allows for focus group participants to provide verbal consent to participate and requests permission from participants to record the focus group. The remote version of the consent form in Attachment J requests permission to record a participant’s interview.
The contact person for participants enrolled in the Tribal Evaluation that is listed on the consent form was changed from Michael Meit (former Tribal Evaluation Project Director) to Carol Hafford (new Tribal Evaluation Project Director). These changes were made across all consent forms for the tribal evaluation (Attachment I, Attachment J, Attachment B2, and Attachment B3).
A2.2 Research Questions
This non-substantive change request does not require any changes to the research questions included in prior versions of the Supporting Statement, nor to this section of the Supporting Statement. The third revised request for clearance, approved in July 2019, covered the research questions applicable to the National Evaluation impact evaluation—to be addressed by the Intermediate Follow-up Survey and the phone-based Skills Assessment Pilot; as well as the additional data to be collected in support of the descriptive evaluation and the cost-benefit analysis study.
The research questions for the National Evaluation descriptive evaluation and the Tribal Evaluation were summarized in a previously approved request for clearance, along with their respective data collection protocols (OMB Control Number 0970-0462, approved June 2017). The research questions from that prior submission are included in Attachment O.
Exhibit A-2 provides a schematic for the theory of action for the HPOG 2.0 National Evaluation’s impact evaluation. The top row of the exhibit represents the experiences of applicants randomized to the treatment group—that is, those offered a “slot” in an HPOG 2.0 program, where “slot” means the package of training and associated support services offered by the program, whether or not the individual uses the components of that package. Conversely, the bottom row represents the experiences of those in the control group, who are not offered an HPOG 2.0 slot.
Exhibit A-2: Schematic and Theory of Action of the HPOG 2.0 Impact Evaluation
From left to right in the top row of the exhibit, an applicant randomly assigned to the treatment group is offered an HPOG 2.0 slot and thereby gets access to the training and associated support services from the HPOG 2.0 program and, potentially, from other sources. (Nevertheless, but not explicitly shown in the exhibit, not everyone offered access to HPOG services will use everything—or even anything—offered.) The hypothesis to be tested is that HPOG’s package of training and associated support services leads to impacts on educational and occupational credentials, employment, working conditions (hours, hourly wage, shift work, benefits), and earnings. Impacts on public assistance receipt (TANF, SNAP, Medicaid, and unemployment insurance) and broader aspects of well-being (food security, housing stability, and marital status) may also emerge.
In contrast, those randomly assigned to the control group (the bottom row of the exhibit) are not offered access to the HPOG 2.0 program, but may obtain training and other support services from other sources. The same set of outcomes emerges, though possibly at different levels and with different timing: education and credentials, employment and earnings, public assistance, and overall well-being.
Though not everyone in the treatment group gets training and many in the control group do get training, the two contrasting flows in Exhibit A-2 represent the very contrast relevant to future policy decisions on funding HPOG-like services. Random assignment creates a treatment group and a control group that differ only by the offer of HPOG 2.0 and chance. Because the two groups are otherwise statistically equivalent, comparisons of outcomes between them provides an unambiguous estimate of the impact of HPOG 2.0; “impact,” refers to outcomes for those offered HPOG in a world with the program relative to what outcomes for those same individuals would have been had HPOG not existed.
The National Evaluation considers all aspects of the theory of action shown in Exhibit A-2:
Addressing implementation research questions, the descriptive evaluation describes the HPOG 2.0 program as implemented.
Addressing service contrast research questions, the service contrast analysis estimates the impact of the offer of HPOG 2.0 on services received—training and other support services.1
Addressing impact research questions, the impact analysis estimates the impact of the offer of HPOG 2.0 on outcomes of interest—including educational programs completed, credentials received, employment, earnings, and participation in public assistance programs.
Assessing the costs of implementing the program relative to the benefit to participants and grantees is the primary goal of the cost-benefit analysis.
Exhibit A-3 provides the research questions that pertained to the data collection activities included in the third revised request for clearance, approved in July 2019.
Exhibit A-3: Research Questions Relevant to the Third Revision Approved in July 2019
Evaluation Component |
Data Source2 |
Research Questions |
Descriptive Evaluation |
Second-round telephone interviews |
|
Participant In-depth interviews |
|
|
Systems study telephone interviews with program operators and partner organizations |
|
|
Impact Evaluation |
Intermediate Follow-up survey |
Research questions on the impact of being offered an HPOG 2.0 slot:
The research questions above are framed as the impact of being offered a slot. If sufficient resources are available, the evaluators will address an additional research question:
|
Skills Assessment Pilot |
The Skills Assessment Pilot is intended to determine experimentally if programs such as HPOG 2.0 improve the kinds of basic skills deemed most relevant to qualifying students to participate in education and training programs designed to prepare them for healthcare occupations. The pilot will help to identify a parsimonious subset of items that are relevant to success in healthcare training and that achieve the right mix of level of difficulty, as determined though item response modeling (IRT). It will also determine if these skills can be assessed over the telephone. Once identified, this subset of items will be incorporated into the Intermediate Follow-up Survey as a separate module at the end, in order to address this research question:
|
|
Cost-benefit analysis study |
Program cost survey |
|
A2.3 Study Design
This section provides an overview of the design of the HPOG 2.0 National Evaluation. For an overview of the Tribal Evaluation study design, please see the first revision under OMB Control No. 0970-0462 approved in June 2017. For more detail on the previously approved PAGES system, please see the original submission under OMB Control No. 0970-0462 (approved in August 2015). PAGES was designed to meet the performance data needs of the grantees and of OFA to monitor the grantee performance and prepare the report to Congress on the grants. PAGES supports the National and Tribal Evaluations, as well as other future research and evaluation efforts sponsored by ACF.
HPOG 2.0 National Evaluation Study Design
The HPOG 2.0 National Evaluation is guided by the career pathways framework, as shown in the HPOG logic model (Attachment H). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012). These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports and employer connections.
The design for the HPOG 2.0 National Evaluation features a
descriptive evaluation (including an implementation study, a systems study, and an outcome study);
an impact evaluation (using a classic experimental design to measure and analyze key participant outcomes including completion of education and training, receipt of certificates and/or degrees, earnings, and employment in a healthcare career); and
a cost-benefit analysis study.
Exhibit A-4 provides a visual description of the major components and sub-components of the HPOG 2.0 National Evaluation.
Exhibit A-4: Components of the HPOG 2.0 National Evaluation
Briefly, and as discussed above, the impact evaluation design includes randomizing program-eligible participants to treatment and control status in all non-tribal sites. Follow-up to answer the research questions will involve both queries of administrative data systems and participant surveys. The surveys are described in more detail in Section A4 below. The research team will match participant data collected through the impact evaluation for both the treatment and control groups to long-term employment and earnings data from ACF’s National Directory of New Hires (NDNH) and to school enrollment data from the National Student Clearinghouse (NSC). An agreement with the Office of Child Support Enforcement (OCSE) to use the NDNH data was signed in February 2018. Negotiations to use the NSC data are underway.
HPOG 2.0 Tribal Evaluation Study Design
The HPOG 2.0 Tribal Evaluation is designed as a comprehensive implementation and outcome evaluation.3 The approach for the evaluation is guided by the seven values outlined in the Roadmap for Collaborative and Effective Evaluation in Tribal Communities, developed by the Child Welfare Research and Evaluation Tribal Workgroup.4 The values provide guidance for partnering with tribal communities and are grounded in community-based participatory research. All five tribal grantees will participate in the HPOG 2.0 Tribal Evaluation. The evaluation will use a mixed-methods approach, including collection of qualitative data through interviews and focus groups and analysis of program documentation and program data. Qualitative data will be collected during annual site visits to each of the five Tribal HPOG 2.0 grantees. When annual site visits are not possible, qualitative data will be collected via telephone or virtual meeting.
A2.4 Universe of Data Collection Efforts
To address these research questions, the HPOG 2.0 National Evaluation will use a number of data collection instruments. Exhibit A-5 describes the target respondents, content, and reason for collection (i.e., which analyses will use the information) for each of the data collection activities submitted with the third revised submission request and approved in July 2019. All other support materials (for example the survey advance letter or the survey flyers) for the new instruments are provided in Attachments P, R, S, T, U, V, W, and Z.
For a list of study instruments approved by OMB in prior information collection requests, see Attachment X.
Exhibit A-5: HPOG 2.0 National Evaluation Instrument Overview
Data Collection Activity |
Data Collection Instrument(s) |
Respondents, Content, and Reason for Collection |
Descriptive Evaluation Second-Round Telephone Interviews (COMPLETE) |
Screening Interview to identify respondents for the HPOG 2.0 National Evaluation second-round telephone interviews (Instrument 13)
HPOG 2.0 National Evaluation second-round telephone interview guide (Instrument 14) |
Respondents: Program staff, managers, partners, and stakeholders at the 38 programs administered by the 27 non-tribal grantees. The telephone interviews took place in Fall 2019.
Content:
Reason: The screening interview was conducted over the phone with HPOG program managers to identify appropriate respondents for the second-round telephone interviews. The second-round telephone interviews built on the first round of interviews (previously approved in June 2017 under OMB Control No. 0970-0462). These interviews collected information about the HPOG program context and about program administration, activities and services, partner and stakeholder roles and networks, and respondent perceptions of the program’s strengths. The second round placed additional focus on implementation and performance successes and challenges; documentation of changes since the start of the grant; and information on how grantees spend their grant funds. Information gathered during interviews inform the descriptive evaluation’s implementation study. These data are not available through any current sources. |
Descriptive Evaluation Systems Study Interviews (COMPLETE) |
HPOG 2.0 National Evaluation systems study interview guides (Instruments 15 and 16) |
Respondents: Program staff and partners at a subset of purposively selected non-tribal HPOG programs (12-16 of the 38 programs). The telephone interviews occurred in fall 2019.
Content:
Reason: Systems study interviews explored whether and how the HPOG 2.0 grants supported systems activities for providing healthcare training opportunities to low-income adults. The interviews were the key data source for the systems study, which builds on findings about the program operations and partnerships described by the Implementation Study to learn from HPOG programs and the systems in which they operate. These data are not available through any current sources. |
Descriptive Evaluation Participant Interviews (COMPLETE) |
HPOG 2.0 participant interview guide (Instrument 17) |
Respondents: Expected sample of 140 treatment group members, across 14 HPOG programs. The in-person interviews we conducted in fall 2019 through early winter 2020.
Content:
Reason: The goal of these in-depth qualitative interviews was to gain insights into the motivations, decision making, expectations, and experiences of HPOG 2.0 Program participants. These data are not available through any current sources. Many of the questions asked in the interviews were based on questions approved for ACF’s Pathways for Advancing Careers and Education participant interviews (OMB #: 0970-0397) to offer continuity in the information collected across ACF’s career pathways research portfolio. |
Impact Evaluation Participant Follow-up Survey 36 months after randomization (ONGOING) |
Intermediate Follow-up Survey for the HPOG 2.0 National Evaluation (Instrument 18, and new version Instrument 18a) |
Respondents: Overall expected sample of 5,000 (selected participants randomized between March 2017 and February 2018). Data collection began in September 2020 (by telephone and then in-person for those respondents who cannot be reached by telephone).
Content:
Reason: The Intermediate Follow-up Survey will collect information on events that have occurred since the Short-term Follow-up Survey (approximately 15-months post-randomization) in many areas—particularly the receipt of training and related supports, and receipt of credentials. This survey information will provide outcomes for the impact analysis. These data are not available through any current sources. Many of the questions to be asked in this survey were approved for the Short-term Follow-up Survey for the HPOG 2.0 National Evaluation (OMB Control No. 0970-0462), along with other studies in ACF’s Career Pathways portfolio, specifically the Pathways for Advancing Careers and Education 15-, 36-, and 72-month follow-up surveys (OMB #: 0970-0397); and the impact study of the first round of HPOG 15-, 36-, and 72-month surveys (OMB #: 0970-0394). (A summary of the 36-month survey item sources is provided in Attachment Q.) ACF seeks approval to add a new, shorter version of the Intermediate Follow-up Survey (Instrument 18a) that collects data on just the critical items of interest. This shorter version would be used as a tool to maximize response rates and deal with nonresponse. Note: The revised Intermediate Follow-up Survey now includes just a subset of items selected after analysis of a pilot test of the skills assessment module (Instrument 19) was completed. |
Impact Evaluation phone-based Skills Assessment Pilot (COMPLETE) |
Questionnaire for a phone-based Skills Assessment Pilot (Instrument 19) |
Respondents: Expected sample of 500 HPOG 2.0 participants. Non-tribal grantees will be asked to help identify potential volunteers to participate. The phone-based skills assessments were completed in late fall 2019.
Content:
Reason: The logic models for career pathways programs typically include a path toward increasing earnings. Accordingly, it is important to determine experimentally if programs such as HPOG 2.0 improve the kinds of basic skills deemed most relevant to qualifying students to participate in education and training programs designed to prepare them for healthcare occupations. Since telephone interviews with study subjects almost always play an important role in experimental evaluations of these programs, it would be useful to have a module that measures program participants’ level of these basic skills over the phone. The mode and burden of collecting this type of basic skills information with other methods—in-person or online—can be quite expensive and burdensome on participants, making these methods infeasible for a large-scale evaluation like HPOG 2.0 National Evaluation. The phone-based Skills Assessment Pilot administered a series of 45 literacy and numeracy based assessment questions in order to identify a shorter subset for inclusion in the Intermediate Follow-up Survey. The final subset recommended for inclusion in the Intermediate Follow-up Survey will be used to assess respondents’ literacy and numeracy experimentally. Respondents will only be asked up to 16 of the 22 questions included in the survey, eight in each category. The final set of items are included in Section J of the revised Instrument 18. These data are not available through any current sources. |
Cost-benefit analysis study Program cost data collection (COMPLETE) |
HPOG 2.0 Program Cost Survey (Instrument 20) |
Respondents: Staff at the 38 non-tribal HPOG programs. The cost surveys were completed between fall 2019 and spring 2020.
Content:
Reason: The purpose of the cost survey was to gather information on HPOG 2.0 program costs, including expenditures on staff, overhead, academic supports, employment supports, and other supports. The Cost-benefit analysis study will use results of this survey to estimate costs per participant for academic and employment supports, assistance with transportation and training-related materials (including school fees, supplies, uniforms, testing fees, equipment and tools), as well as to measure the costs of program administration. These data are not available through any current sources. |
Other extant data sources will be used for the HPOG 2.0 National and Tribal Evaluation. These include the following:
National Directory of New Hires (NDNH). These data will provide information on employment and earnings of HPOG participants.
National Student Clearinghouse (NSC). These data will provide information on student enrollment in credit-bearing courses (and some enrollment in non-credit bearing courses) and receipt of post-secondary degrees.
HPOG program management information, including initial applications and ongoing management reports, which will provide supplemental information in tracking the evaluation of the grant, and information on the local healthcare labor market and needs for occupational training.
Government sources of labor market data, from the U.S. Census Bureau and Bureau of Labor Statistics (BLS), such as County Business Patterns, Local Area Unemployment Statistics (LAUS), and Quarterly Workforce Indicators (QWI), which will provide a picture of the local labor market.
A3: Improved Information Technology to Reduce Burden
The HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluations will generate a substantial amount of data using a combination of data collection methods. The evaluation team designs each data collection protocol to limit the reporting burden for respondents. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. As described in the originally approved supporting statement (approved in August 2015, with revisions in January and July 2016, June 2017, and June 2018), participant- and grantee-level data will be collected through PAGES, a cloud-based data system. The evaluation teams will use the quantitative data collected through PAGES to reduce respondent burden wherever possible. The team will rely on data collected during the first-round telephone interviews (see Section A4 for more detail) to pre-populate the second-round interviews where possible. The team will also rely on administrative data—such as NDNH—to capture employment and wage data. This removes the burden of collecting this information from participants during the follow-up survey. Any requests for program documentation will be collected electronically as well.
The HPOG 2.0 National Evaluation impact evaluation will offer study participants the option to update their contact information online, by mail, or by telephone. The follow-up surveys will be administered using computer assisted personal interviewing (CAPI) technology for all interviews. CAPI technology reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns are properly implemented, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who did not participate in postsecondary training will be routed past questions only relevant to those who did. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.
The HPOG 2.0 National Evaluation impact evaluation will use improved information technology to pilot a new way to assess basic skills. Numerous tools to measure basic skills already exist, but most require in-person or online data collection capabilities. In-person assessments are often too costly for most large-scale studies, and they put more burden on respondents. Online assessments are not always feasible for large-scale studies with populations similar to the HPOG 2.0 National Evaluation as some participants may not have computer or internet access. The phone-based Skills Assessment Pilot, completed in Fall 2019, allowed the evaluation team to identify a short battery of questions—11 literacy and 11 numeracy questions, 22 in total—for conducting skills assessments by telephone. This battery of questions is included in Section J of the revised version of Instrument 18. The participants will initially be asked five questions in the literacy scale and five questions in the numeracy scale of medium difficulty. Depending on the number of correct responses to the five medium difficulty questions, CAPI technology will display either three questions from the easier or three from the more difficult questions. Phone-based methods to assess basic skills offers an alternative that reduces the costs to the Federal Government and reduces the burden on the respondents.
The HPOG 2.0 Tribal Evaluation team will request permission from participants to record all focus group and participant interviews conducted either by phone or virtually. A recorded interview allows the interviewer to remain engaged with the respondent and helps ensure efficient interview administration. Rather than the interviewer pausing to make sure they type or write down all of the responses verbatim, they will always be able to refer back to the recorded responses for analysis and report-writing.
A4: Efforts to Identify Duplication
Abt and their partners are also conducting several other evaluations on behalf of ACF as part of the HPOG research portfolio on the first round of HPOG grantees, for which there are several data collections already approved by OMB (see Attachment F for further details). ACF and its contractors are engaged in many efforts to coordinate these evaluation activities and avoid duplication of work. The HPOG 2.0 Evaluation team has used the extensive knowledge generated to date from the research activities on the first round of HPOG and ACF’s Pathways for Advancing Careers and Education (PACE) programs (OMB control numbers 0970-0394 and 0970-0397, respectively) to inform the proposed new data collection efforts for the second round of HPOG grantees. This section summarizes those efforts for the National Evaluation impact evaluation, descriptive evaluation, and cost-benefit analysis study.
Impact Evaluation. The purpose of the HPOG 2.0 National Evaluation’s Intermediate Follow-up Survey is to obtain current information on the status and wellbeing of individuals in the HPOG 2.0 National Evaluation study sample members 36 months after randomization. Information about these respondents' educational achievement, economic well-being, job skills development and progression, and overall well-being are not consistently available through any other source, nor is information about family composition, student debt, or 21st century skills. The evaluation will utilize administrative data (e.g., wage records) in conjunction with survey data to avoid duplication of reporting.
The research team will also avoid duplication in this study by use of a study-specific database, maintained by Abt, which links all the data collected at baseline with subsequent information gathered from future surveys and administrative sources. This eliminates the need to ask about personal characteristics or background factors for known household members on follow-up surveys.
The phone-based Skills Assessment Pilot survey collected data from HPOG participants that were not available through any other source. Although HPOG 2.0 grantees may conduct their own basic skills assessments at intake, these assessments cannot substitute for this information collection. First, the assessment protocols that the programs use are quite diverse across grantees. Second, these assessments are generally only administered prior to randomization and thus are not available to examine effects of the HPOG Program. Finally, the exact scores of assessments done at enrollment are mostly missing in PAGES because they are not required fields. To address this issue, our proposed approach has the potential to identify a short subset of questions that can be incorporated into the Intermediate Follow-up Survey. This subset of questions will help to assess basic skills of HPOG participants and thereby be informative about the adequacy of preparation of HPOG 2.0 participants for the early phases of their healthcare career.
The pilot also tested the feasibility of telephone administration for this type of assessment. Several national and international surveys have been developed to assess adult numeracy and literacy, but almost all of these rely on face-to-face interviewing or online administration. Since the Intermediate Follow-up Survey will be administered over the telephone and in-person, the phone-based Skills Assessment Pilot will help to ensure that a trained interviewer can administer the assessment questions by telephone, making it appealing for HPOG 2.0 and potentially other OPRE evaluations as well. The recommendations from the now completed pilot study analysis are reflected in this non-substantive change request.
Descriptive evaluation. The purposes of the HPOG 2.0 National Evaluation second-round telephone interviews, the telephone interviews for the systems study, and the participant interviews were to obtain information about HPOG grantee programs, partners, and participants that was not available through any other source. Wherever possible, the research team used existing sources of information—including PAGES, available information from the site-specific evaluation design and other existing site-specific materials developed earlier by the National Evaluation team—to collect key information prior to conducting the telephone interviews. These existing data sources and information available from each include the following:
Evaluation Design and Other Site-Specific Materials
Target population and recruitment strategies
Eligibility criteria and application processes
Control group services and conditions
PAGES and Program Performance Progress Reports (PPRs)
Available training courses and support services
Participant characteristics
Participant take-up of training courses and support services
Participant education and employment outcomes
Site Team Monitoring Reports
Documented program changes
Grantee applications
Organization type
Cost-benefit analysis study. The purpose of the cost data collection was to obtain information about HPOG grantees program costs that is not available through any other source. The data collected will complement other data sources, not duplicate or replace them. The research team will use other existing sources of information to estimate costs, including the following:
Delta Cost Project Database (DCPD): A publicly available longitudinal database derived from the U.S. Department of Education’s Integrated Postsecondary Education Data System (IPEDS). The database translates IPEDS information into analytic formats for analysis of revenues and expenditures in postsecondary education.
Follow-up Surveys of Study Participants: The Short-term (15-month) Follow-up Survey approved in June 2018 and the Intermediate (36-month) Follow-up Survey approved in July 2019, with revisions approved in June 2020.
PAGES: Data from programs on participant receipt of training and services funded by their grants.
A5: Involvement of Small Organizations
The National Evaluation and Tribal Evaluation will have minimal impact on small organizations. The primary organizations involved in this study are tribal and community colleges, workforce development agencies, tribal organizations, and community-based organizations that operate occupational training programs. The funding announcement informed all grantees of the federal evaluation and reporting requirements, and adequate resources have been provided to coordinate the data collection and reporting. There should be no adverse impact for any grantees participating in the study.
For the descriptive study, small business professionals will only be interviewed if they are HPOG grantee partners or employers of HPOG program graduates. Interviewers will make every effort to complete these interviews as efficiently as possible.
There is no small business involvement in the other data collection efforts included in this submission.
A6: Consequences of Less Frequent Data Collection
This section summarizes the consequences of less frequent data collection for the HPOG 2.0 National Evaluation impact evaluation, descriptive evaluation, and cost-benefit analysis study.
Impact evaluation. For the HPOG 2.0 National Evaluation impact evaluation, ACF is considering one round of substantive data collection with individual participants enrolled during the COVID-19 pandemic. If funded, the evaluation contractor will interview this cohort at 15 months following randomization. Instruments 5a and 5b will allow the evaluation contractor to request periodic updates to the respondent’s contact information. Updated contact information is vital to the success of the COVID-19 cohort survey, if it is funded. ACF and the evaluation contractor are developing the COVID-19 survey instrument and procedures and will submit them for OMB review separately as part of a full information collection request, with additional time for public comment.
Descriptive evaluation. The evaluation team planned two rounds of telephone interviews with HPOG grantees. The first round of interviews took place in year 2 of the grant. The second round took place in year 4 of the grant. The descriptive evaluation will include only one round of participant interviews and telephone interviews with HPOG partners for the systems study. Less frequent data collection would prohibit timely collection of data about program implementation, costs, and systems over time.
Cost-benefit analysis study. The evaluation team plans only one round of cost data collection from HPOG grantees, completed between September 2019 and March 2020. This data collection is critical to estimating the cost of program inputs for the cost-benefit analysis study.
A7: Special Circumstances
There are no special circumstances for the proposed data collection.
A8: Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 27, 2018, Volume 83, Number 247, page 66715-66717, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment Y. During the notice and comment period, the government received no requests for information or substantive comments.
Consultation with Experts Outside of the Study
The HPOG 2.0 National Evaluation team had limited consultation with external experts in developing the instruments previously approved under OMB# 0970-0462. The design of the National Evaluation Intermediate Follow-up Survey is based on the previously approved Short-term Follow-up Survey (OMB Control No. 0970-0462 approved in June 2018), along with other studies in ACF’s career pathways research portfolio, specifically the PACE 15-, 36-, and 72-month follow-up surveys (OMB #: 0970-0397); and the impact study of the first round of HPOG 15-, 36-, and 72-month surveys (OMB #: 0970-0394) to ensure consistency in the data collected across ACF’s entire career pathways research portfolio.
The external consultation focused primarily on how to measure basic skills, which is the focus of the phone-based Skills Assessment Pilot (Instrument 19). The experts consulted are listed in Exhibit A-6 below. This consultation took place in 2017.
Exhibit A-6: Experts Consulted Outside of the Study
Name |
Title/ Organization |
Contact Information |
HPOG 2.0 National Evaluation |
||
Meredith Larson |
Research
Analyst U.S. Department of Education |
[email protected] |
Stephen Provasnik |
Team Lead for International Activities Program National Center for Education Research Institute of Education Sciences U.S. Department of Education |
[email protected] |
Irwin Kirsch |
Director of the Center for Global Assessment Education Testing Services |
1-609-921-9000
|
The previously approved PAGES data items and the Tribal Evaluation materials were also developed in conjunction with substantive experts. The ICRs previously approved under OMB Control No. 0970-0462 in 2015 and 2017, respectively, provide more information on external consultation for those items. The instruments for the descriptive evaluation and cost-benefit analysis study did not require consultation with experts. The evaluation team includes staff nationally recognized for their expertise in cost-benefit analysis (Bob Lerman and Pam Loprest from The Urban Institute, Mary Farrell from MEF Associates). These team members also consulted with some of the non-tribal grantees to solicit their feedback on the content and burden associated with the cost survey.
A9: Incentives for Respondents
The evaluators plan to offer incentives to respondents for the National Evaluation impact and descriptive evaluations. Specifically, incentives are planned for the Intermediate Follow-up Survey, the phone-based Skills Assessment Pilot, and the participant in-depth interviews. The justification and incentive plans for each data collection activity are provided below.
Incentives—National Evaluation
Monetary incentives show study participants that the researchers appreciate their continued involvement in the HPOG 2.0 National Evaluation information collection activities. The HPOG 2.0 National Evaluation impact evaluation is a longitudinal panel RCT study, intended to follow selected impact evaluation participants for at least three years. Although there is little published evidence of the effectiveness of incentives in reducing nonresponse bias, it is well established that incentives strongly reduce study attrition (i.e., increase response rates) in panel studies such as the HPOG 2.0 National Evaluation.5,6 In accordance with OMB guidelines; the team took several factors into consideration when determining whether or not to use incentives.7 OMB published guidance to help agencies ensure that they maximized the quality of information collected.8,9 OMB defined “quality” in that guidance as “…the encompassing term, of which ‘utility,’ ’objectivity,’ and ‘integrity’ are the constituents.” Utility refers to how useful the information is to the intended audiences; objectivity focuses on both the presentation of the information collected and the substance of the information collection methods; and integrity takes into account the information collection protocols particularly as they relate to data security. This was taken into consideration when determining whether or not the use of incentives was necessary for the information collections that are the subject of this third revision of OMB Control Number 0970-0462.
The HPOG 2.0 National and Tribal Evaluation will ultimately provide ACF with information necessary to help objectively assess the HPOG Program and inform key stakeholders about lessons learned. It will provide policy makers with information to help make decisions about future reauthorization of the Program. The evaluation team and ACF worked closely to ensure that the previously approved procedures related to the data collected under this OMB Control Number met the OMB requirements for quality and transparency. Quality—from the perspective of integrity—is addressed in Section A10; while plans to ensure quality—with regard to objectivity and utility—are covered throughout Supporting Statement A and Supporting Statement B.
With regard to the use of incentives, the team focused on ensuring quality—particularly quality with regard to objectivity. The team took into account the key data quality risks posed by the longitudinal study design, our other efforts to reduce non-response bias, the burden on respondents, the complexity of the study design and panel retention over a 36-month period, and prior use of incentives for this study population previously approved under this OMB Control Number.
In a panel study such as the HPOG 2.0 National Evaluation, panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving sufficient sample size to maintain statistical power to detect meaningful effects in the analysis. Although low response rates do not necessarily lead to nonresponse bias, and it is at least theoretically possible to increase nonresponse bias by employing some techniques to boost response rates (Groves, 2006), most statisticians and econometricians involved in the design and analysis of randomized field trials of social programs agree that it is generally desirable to obtain a response rate close to 80 percent in all arms of the trial (Deke and Chiang, 2016). Based on the research team’s experience with differential response rates in the PACE and HPOG 1.0 data collection efforts—studies of similar scope and populations, the team believes that there is some risk that the HPOG 2.0 study will be sensitive to differential response rates, particularly if the team does not minimize attrition between the Short-term (15-month) and Intermediate (36-month) Follow-up Surveys. PACE had a differential response rate of 5.1 percentage points at 15-months and 5.0 percentage points at 36-months. In the HPOG 1.0 three-armed experiment, the response rate differential for the standard treatment vs. the control group was 7.2 percentage points and 6.6 percentage points for the enhanced treatment vs. the control group at 15 months. At the end of the 36-month follow-up data collection, the response rate differential was 3.4 percentage points between the standard treatment and control group, and 5.9 percentage points between the enhanced treatment and control group.
The treatment and control differential improved between the 15- and 36-month follow-up surveys for both PACE and HPOG 1.0 Impact studies. The team theorizes that incentives were a particularly powerful tool for maintaining a high response rate in the control group given that these sample members do not receive any (other) program benefits or services. Given the similarities between these studies, we expect similar response patterns for HPOG 2.0.
In most panel studies, response rates decline over follow-up rounds, potentially weakening the quality of the information collection—with regard to objectivity—as lower response rates could lead to increased nonresponse bias. The team has tried to minimize this expected decline and ensure a high response rate with a low treatment-control differential through the use of the previously approved welcome packet and repeated use of the previously approved participant contact update forms, combined with the provision of incentives, as discussed below. Through these tools the team hopes to address three goals:
Overcome participant mobility—over a long follow-up period, many study participants relocate multiple times, making it difficult to find them to complete a follow-up interview;
Reduce survey data collection costs— the more quickly interviewers can locate the respondent and complete an interview, the lower the costs per completed survey; and
Maintain participant engagement in a complex panel study—the ability to keep participants engaged in the research study for at least three years after enrollment is crucial to understanding long-term outcomes and the effectiveness of the HPOG Program.
The National Evaluation also provided incentives to those who participated in the in-depth interviews for the descriptive evaluation. The in-depth interviews were long and they often required participants to make arrangements for child care or transportation. The use of incentives to help offset the costs of participation is a well-established practice in social science research and program evaluation for both small-scale studies and sample surveys.
Previously Approved Incentives—National Evaluation
OMB previously approved the use of incentives in support of this study—the HPOG 2.0 National and Tribal Evaluations—under two prior revisions to OMB Control No. 0970-0462 (approved in June 2017 and June 2018). We drew upon experience with the incentives approved for the prior data collection components to specify the appropriate incentive amount for the additional HPOG 2.0 National and Tribal Evaluation data collection components proposed and approved under the third modification request.
Exhibit A-7 provides a summary of the incentives approved under the HPOG 2.0 National and Tribal Evaluation prior to the third modification request. Incentives approved in July 2019, under the third modification request, are summarized below.
Exhibit A-7: HPOG 2.0 National and Tribal Evaluation Previously Approved Incentives
Data Collection Activity |
Data Collection Instrument(s) |
Incentive Amount |
HPOG 2.0 National Evaluation |
||
Impact evaluation |
Welcome Packet (Instrument 5a, approved in June 2017) |
Participants received a non-monetary incentive (a portable cell phone charger), branded with the HPOG 2.0 study logo and toll-free study hotline phone number. |
Contact Update Request (Instrument 5b, approved in June 2017) |
The evaluation team provides an incentive—a $5 gift certificate—for each contact update received from participants. |
|
Short-term (15-month) Follow-up Survey (Instrument 12, approved in June 2018) |
Respondents received a gift card valued at $40 upon completion of the survey. |
|
HPOG 2.0 Tribal Evaluation |
||
Tribal Evaluation |
Focus Group Guides
HPOG Program Completer Interviews
HPOG 2.0 Program Non-completer Interviews (Instruments 9, 10, and 11, approved in June 2017) |
Respondents receive a non-cash incentive valued at $50. |
Incentives Approved in July 2019 under the Third Revised Request for Clearance—National Evaluation
Impact evaluation. In the third revised request for clearance, approved in July 2019, the National Evaluation team requested and received permission to provide incentives for completion of the Intermediate Follow-up Survey and the phone-based Skills Assessment Pilot.
Three factors informed the study’s choice of the incentive amounts for survey respondents:
Respondent burden, both at the time of the interview and over the life of the study;
Costs associated with participating in the interview at that time; and
Other studies of comparable populations and burden—to help with the estimation of the appropriate incentive amount.
Given a target response rate of 80 percent for the
Intermediate Follow-up Survey, a minimum three-year follow-up period,
the incentive amounts previously approved for the HPOG 2.0 National
Evaluation impact evaluation Short-term Follow-up Survey (OMB control
number 0970-0462), and a similar study design and population in
OPRE’s Career Pathways studies (PACE and the first round HPOG
Impact studies; OMB control numbers 0970-0397 and 0970-0394
respectively), the evaluation team believes that incentives are
necessary to maximize the response rate to the impact evaluation’s
Intermediate (36-month) Follow-up Survey, balance the treatment and
control group response rate differential, and minimize attrition.
The impact evaluation team believes that an incentive of
$45 is the appropriate amount to help ensure that the evaluation
ultimately meets the quality targets defined by OMB. The Intermediate
Follow-up Survey sample is smaller than that for the Short-term
Follow-up Survey, so minimizing attrition is critical. Therefore, the
proposed incentive amount represents a modest $5 increase— from
$40 at the Short-term Follow-up Survey to $45 at the Intermediate
Follow-up Survey. These incentives are provided to help offset any
potential expenses incurred by the participant such as cell phone
minutes for those completed by telephone or childcare/transportation
costs for those completed in-person. As approved in July 2019, the
Intermediate Follow-up Survey respondents will receive a gift card
valued at $45. Respondents will receive an email with instructions to
log in to a secure study portal where they can redeem a $45 gift card
from their choice of approved vendors. 10
Respondents who complete the interview in person and do not have an
email address will receive a card with redemption instructions.
A similar approach was used for the phone-based Skills Assessment Pilot. The evaluation team provided incentives to respondents who participated in this pilot data collection activity. Given that this was an exploratory effort, the length of the data collection activity was somewhat shorter and less burdensome to respondents than the follow-up surveys, and the participants who completed the assessment volunteered to do so, the evaluation team believed that an incentive valued at $25 was sufficient. Participants who completed the assessment received a gift card valued at $25 (to be redeemed in the same manner as described above). The evaluation team ensured that the participants who volunteered to participate in the assessment pilot were not part of the follow-up survey samples.
Following completion of the Short-term Follow-up Survey, a participant will continue to receive quarterly contact update requests in preparation for the Intermediate Follow-up Survey. Participants will receive the requests to update their contact information using the previously approved contact update form (Instrument 5b, approved in June 2017). Those who respond to the quarterly contact update requests will receive the $5 electronic gift certificate after each completed request (see OMB Supporting Statement A for OMB Control # 0970-0462, previously approved in June 2017 for details).
Descriptive evaluation. For the descriptive evaluation, the research team requested and received approval under the third revised submission to provide non-cash incentives for in-person participant interviews, in the form of a gift card. These interviews are estimated to take between 60 and 90 minutes to complete. Interview data are not intended to be representative in a statistical sense, in that they will not be used to make statements about the prevalence of experiences in the overall HPOG population. However, it is important to recruit participants with a range of background characteristics, to capture a variety of possible experiences with HPOG services. As all participants will be current or former HPOG program participants, the target population is, by definition, low income.
Although participation is voluntary and participants will have a choice of travelling to an interview or having an interviewer travel to them, there are potential burdens placed on study participants. These interviews may impose burden on the participant’s daily life. For instance, an interview that takes 60-90 minutes to complete could interfere in family commitments, result in substantial travel or child care costs, or result in unforeseen other expenses associated with research participation. Without offsetting the direct costs incurred by respondents for attending the interviews, such as arranging child care, transportation, or time off from paid work, the research team increases the risk that only those individuals able to overcome the financial barriers to attend will participate in the study, thus limiting the experiences the study is able to capture. For example, this may result in fewer participants with children participating in the study and thus the study would not reveal the experiences of attending the HPOG program while raising children.
Participants received a $40 gift card to account for expenses such as transportation and/or child care that may otherwise prevent their participation in the study. Studies have shown incentives’ effectiveness in increasing study participation among underrepresented populations such as individuals from low-income or low-education households, demographics of concern here. The amount of incentive is important in increasing participation rates. While research suggests little difference in the incentive amount when a participant is willing to participate, for participants who are less willing to participate, a larger incentive amount increases the likelihood of the potential participant agreeing to conduct the interview.11 Within the HPOG 2.0 National and Tribal Evaluation, OMB previously approved similar use of incentives for the HPOG 2.0 Tribal Evaluation (participant focus groups and interviews) in June 2017 under this OMB Control Number (0970-0462) in the amount of $50.The ACF PACE study (OMB control number 0970-0397, approval 08/19/2013), conducted with a similar population to that targeted by the current study, provided $40 incentive payments for interviews that lasted 60-90 minutes and 84% of those asked, participated. We believe $40 is a reasonable amount for the time and cost associated with participation in these data collection activities, but is not so high as to appear coercive for potential participants.
To receive the $40 gift card, participants received an email with instructions to log in to a secure study portal where they can redeem the gift certificate to one of the approved vendors (see procedures for redeeming procedures described under the Intermediate Follow-up Survey).
A10: Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
For all the non-tribal grantees participating in the national evaluation, study participants must provide an SSN in order to enroll in the program. The previously approved consent forms (Attachment B: National Evaluation Informed Consent Form A (Lottery Required) and National Evaluation Informed Consent Form B (Lottery Not Required); and Attachment B2: Tribal Informed Consent Form A (SSNs)) clearly state how SSNs will be used in the evaluation. Those same statements are reflected in the new verbal consent form versions—Attachment B: National Evaluation Informed Consent Form C (Lottery Required)_Verbal and National Evaluation Informed Consent Form D (Lottery Not Required)_Verbal; and Attachment B2: Tribal Evaluation Informed Consent Form C (SSNs)_Verbal.
As specified in the evaluator’s contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. All project and grantee staff members with access to PAGES sign a New User Data Security Agreement and they undergo training on data privacy and security. Grantees participating in the National Evaluation that do not have their own Institutional Review Board (IRB) or Federalwide Assurance Number (FWA) sign individual investigator agreements, which allows the protection under Abt’s FWA. Grantees participating in the Tribal Evaluation that do not have their own IRB or FWA sign individual investigator agreements, which will allow them protections under NORC’s FWA.
As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor has submitted a plan for minimizing to the extent possible the inclusion of sensitive information on paper records (e.g., the consent forms) and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
None of the respondents that participate in interviews will be identified in any report or publication of this study or its results; their participation will be voluntary; and their information will be kept private. This information will be provided verbally to interview respondents in both studies, and verbal consent will be requested.
As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to HPOG 2.0 participants by grantees:
Research is being conducted to see if and how HPOG 2.0 makes a difference in people’s lives by helping them complete training and get healthcare jobs. This program and research are funded by HHS, and HHS may fund other research on this program in the future.
In this program, grantees will collect some personal information from individuals, such as their name, date of birth, Social Security number, and involvement in other programs.12 The researchers studying the program for the government also need this information. Researchers will use data security procedures to keep all of the study data private and to protect individuals’ personal information. All of the information collected for the program or for the research studies will be kept completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
Researchers may contact applicants at grantees participating in the impact evaluation in the future. Individuals may refuse to answer any of their specific questions at any time.
A.10.1 Tribal Evaluation
In June 2017, OMB approved the information collection materials for the Tribal evaluation. As noted in the supporting statements at that time, the Tribal Evaluation focus group participants will receive a written consent form; if data is collected remotely, focus group participants will provide consent verbally. The Program Participant Focus Group Informed Consent (Attachment I) will provide all of the assurances of privacy and data protection information to focus group participants. All Tribal Evaluation interview respondents will provide verbal consent (Attachment J). These consent forms are separate from the consent forms that study participants sign at the time of enrollment.
A.10.2 PAGES
The PAGES system has both a System of Records Notice (SORN)—09-80-0361 OPRE Research and Eval Project Recs—and Authorization to Operate (ATO). For further description of security measures related to PAGES, please see the original submission approved under OMB Control No. 0970-0462 in August 2015.
To ensure data security and enhance data quality, the Intermediate Follow-up Survey data collection will be done using Computer Assisted Personal Interviewing or CAPI technology using the Confirmit CAPI System. The Confirmit CAPI System has the following security features:
Data on the CAPI console is encrypted with Rijndael algorithm (256 bit key).
CAPI data transfers use Web Services Enhancements (WSE 3.0) for security. The messages sent and received from the console are encrypted. WSE 3.0 provides AES128 + RSA 1.5 as default algorithms for symmetric encryption and key-wrap. The research team has also implemented Secure Conversation with an X509 certificate (which uses 1024 bit key).
In addition to the standard security features offered through the CAPI software, the research team has implemented the following enhancements:
Use of PGP whole disk encryption on all CAPI laptops and tablets, and
The file transfers are made to servers running SSL.
As surveys are completed, data will be transferred from the CAPI system to the study’s database. Transfer to the database will be done in a secure manner, using a FIPS-certified encryption algorithm. Once the Intermediate Follow-up Survey data collection is complete, all survey records will be transferred to the analytic database, stored on Abt Associates’ secure Analytical Computing Environment (ACE3), the FISMA moderate server, where most analyses will be conducted. ACE3 currently provides:
A secure, isolated environment utilizing Amazon's FedRAMP Moderate accredited services as infrastructure
Secure server and application configurations that meet NIST SP 800-53 Revision 4 FISMA Moderate standards where appropriate, with compliant policies and procedures
FedRAMP Moderate accredited file transfer services for moving data in and out of the system
Fully redundant architecture where possible, with architected scalability and elasticity to deal with the storage and processing of large data sets by increasing available memory, CPU, or disk space availability
Enhanced monitoring by AWS CloudWatch and the leading third party vendor for Log Monitoring: Dell SecureWorks
Enhanced availability and backups using native AWS services
The analytic databases are designed to limit access only to authorized users with levels of access commensurate with each person’s role on the project. PII will be separated from the rest of the information and stored in a separate folder which only the project director (PD), the deputy project director (DPD), and a small number of designated analysts (5-10 people) will be able to access. The de-identified survey data will be stored in folders that will be accessible by the PD, DPD, PI, the director of analysis, and a small team of other statisticians, economists, and analysts working on the evaluation. Only tabular data and other high-level summaries (such as regression coefficients) will be stored in the general servers of the Prime contractor (Abt Associates), shared with subcontractors via email, and eventually published. The web server hosting the database is maintained in a secure facility with power back up, network redundancy, and system monitoring. In addition, daily back up of the server is maintained at the data center and an off-site location. The database and website are password protected, and access is provided only after user authentication. After the reports are published, the data will be archived. The specific plans for archiving the data and decisions on whether or not to create public or restricted access datasets are to be determined. The archiving procedures will ensure that de-identified data are stored securely.
For participant-level data collected from both survey data and corresponding administrative data from the National Student Clearinghouse (NSC), computer security will be maintained by individual passwords and folder permissions which limit access to files to only those project staff members who require access to these files and have appropriate permission to do so.
All administrative data from the National Directory of New Hire (NDNH) will reside on ACF secure servers. Only Abt staff members granted ACF security clearance will have access to the data on ACF loaned laptops and the secure folder. All analysis of NDNH data will be conducted on ACF’s secure server. All survey information and analytic data will be accessed by the respondent’s study ID number. No personally identifiable information will be maintained on paper.
A11: Sensitive Questions
This section summarizes the sensitive questions asked of respondents under the HPOG 2.0 National Evaluation for the instruments approved in July 2019 and provides guidance on where to find sensitive questions on the other previously approved instruments. It then provides an overview of the sensitive questions contained in the National Evaluation impact evaluation’s Intermediate Follow-up Survey and the National Evaluation descriptive evaluation’s in-depth participant interviews.
PAGES Participant-Level Baseline Questions
For a description of sensitive questions related to PAGES, please see the original submission approved under OMB Control No. 0970-0462.
HPOG 2.0 Tribal Evaluation
For a description of sensitive questions in previously approved Tribal Evaluation instruments, please see the first revised submission approved under OMB Control No. 0970-0462, approved in June 2017.
HPOG 2.0 National Evaluation
Impact evaluation. The questions included in the Short-term Follow-up Survey that could be considered sensitive are described in the previously approved request for clearance under OMB Control Number 0970-0462, approved in June 2018.
The Intermediate Follow-up Survey includes several questions about overall physical health, income, receipt of government benefits, fertility, and household composition, items that some respondents may consider sensitive. As it is hoped that HPOG 2.0 will have favorable impacts in all these areas, failure to ask any of them would limit the findings of the evaluation. Interviewers will remind study members during the interview that they may refuse to answer individual items. Interviewers will also remind study members that their responses will be kept private to encourage their candid responses.
The phone-based Skills Assessment Pilot does not include any sensitive questions.
Descriptive evaluation. Several questions in the HPOG 2.0 National Evaluation participant interviews may be considered sensitive by some program participants. These questions ask about participant and family finances, needs, and types of supportive services received, including academic, social, and employment related. These questions are necessary because supportive services are a key component of the HPOG Program. Data collected will be used to identify how HPOG programs assess student needs and what types of services they are offering as part of their program. Interviewers will inform participants that their participation is voluntary, that they may decline to answer any question that they wish, that their information will be kept private, and they will not be identified in any report or publication of this study or its results. The second round telephone interviews and systems study interviews with HPOG staff, management, and partners do not include any sensitive questions.
Cost-benefit analysis study. The program cost survey will collect only organization-level information and does not include any sensitive questions.
A12: Estimation of Information Collection Burden
The burden estimates described in this section update those approved in July 2019, and revised in June 2020. The non-substantive changes requested in this submission require minor changes to two previously approved estimates: Instruments 5a and 5b, as described below.
This section provides information on the information collection burden estimates in three categories:
The total previously approved burden;
The total burden remaining from the previously approved information collections; and
The total burden increase for previously approved information collections.
A12.1 Previously Approved Information Collections
Total Burden Previously Approved
The previously approved burden estimates included: (1) burden on grantee staff members who enter grantee-level and ongoing participant-level data into PAGES to complete the HPOG PPRs; (2) burden on HPOG applicants to complete the baseline questions; and (3) burden on grantee staff who enter the baseline data into PAGES. It also includes burden for various data collection activities under the National and Tribal Evaluations.
The total burden for all previously approved instruments was estimated to be 20,611 hours annually, or 61,833 hours total over three years.
Burden Remaining from Previously Approved Information Collection
At the time of this nonsubstantive change request (November 2020), a number of information collections are complete. There is no remaining burden for the completed instruments including:
The National Evaluation descriptive study data collection using Instruments 2, 3 and 4 (First-Round Telephone Interview and Screening Guides, and In-person Implementation Interviews) and Instruments 13, 14, 15, 16, and 17 (Second-Round Telephone Interview and Screening Guides, Systems Study program and partner Interview Guides, and In-depth Participant Interview Guides).
The National Evaluation impact study data collection efforts using Instrument 12 (Short-term Follow-up Survey),and Instrument 19 (Phone-based skills assessment pilot); and
National Evaluation cost study Instrument 20.
We have updated the burden table to only include information collections that are still in use.13 Estimated burden remaining to continue use of the previously approved instruments is 8,151 hours annually or 24,453 total hours over the remainder of the approval period. Exhibit A-8 shows the remaining hourly and cost burden by instrument.
Exhibit A-8: Previously Approved Information Collections Still in Use
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Instrument 1: PAGES Grantee- and Participant-Level Data Collection (all grantees) |
96 |
32 |
2 |
31.75 |
2,032 |
$28.29 |
$57,485.28 |
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at non-tribal grantees) |
18,300 |
6,100 |
1 |
.5 |
3,050 |
$3.94 |
$12,017 |
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at Tribal grantees) |
1,288 |
430 |
1 |
.25 |
108 |
$3.94 |
$425.52 |
Instrument 5b: HPOG 2.0 National Evaluation Letter and Participant Contact Update Form |
14,500 |
4,833 |
3 |
.1 |
1,450 |
$10.15 |
$14,716 |
Instrument 18: Intermediate Follow-up Survey for the National Evaluation impact study |
3,785 |
1,262 |
1 |
.92 |
1,161 |
$10.15 |
$11784.15 |
Instrument 18a: HPOG 2.0 National Evaluation Intermediate Follow-up Survey –Critical Items Only |
215 |
72 |
1 |
.33 |
24 |
$10.15 |
$243.60 |
HPOG 2.0 Tribal Evaluation |
|||||||
Instrument 6: HPOG 2.0 Tribal Evaluation Grantee and Partner Administrative Staff Interviews |
70 |
24 |
1 |
1 |
24 |
$28.29 |
$678.96 |
Instrument 7: HPOG 2.0 Tribal Evaluation Program Implementation Staff Interviews |
100 |
34 |
1 |
1.5 |
51 |
$28.29 |
$1,442.79 |
Instrument 8: HPOG 2.0 Tribal Evaluation Employer Interviews |
60 |
20 |
1 |
.75 |
15 |
$50.99 |
$764.85 |
Instrument 9: HPOG 2.0 Tribal Evaluation Program Participant Focus Groups |
270 |
90 |
1 |
1.5 |
135 |
$10.15 |
$1,370.25 |
Instrument 10: HPOG 2.0 Tribal Evaluation Program Participant Completer Interviews |
200 |
67 |
1 |
1 |
67 |
$10.15 |
$680.05 |
Instrument 11: HPOG 2.0 Tribal Evaluation Program Participant Non-completer Interviews |
100 |
34 |
1 |
1 |
34 |
$10.15 |
$345.10 |
Estimated Annual Burden Previously Approved |
8,151 |
|
$ 101,953.55 |
||||
A12.2 Additional Burden for Previously Approved Information Collection
The sample size for the COVID-19 Cohort Survey, if funded, is projected to be 6,400 participants based on grantees’ enrollment projections through September 2021. ACF seeks to increase the number of respondents for the previously approved Welcome to the Study and Contact Update Forms (Instruments 5a and 5b) to begin collecting updated contact information for the COVID-19 Cohort survey study participants. The increased number of respondents will allow the evaluation team to notify participants that they are part of this special cohort that may be selected to participate in an upcoming survey and to send contact update requests to those participants during the 15-month follow up period. Increasing the number of respondents for Instruments 5a and 5b results in a total burden increase of 1,984 hours and adds $20,137.60 to the total costs over the remainder of the approval period. These increases were annualized over the remaining approval period for consistency with other burden estimates under this clearance.
Exhibit A-9 shows the additional burden estimates, in both hours and cost, associated with the increased survey sample size and extended contact update efforts.
Exhibit A-9: Additional Burden for Previously Approved Information Collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Instrument 5a: HPOG 2.0 National Evaluation Welcome to the Study Packet |
6,400 |
3,200 |
1 |
.1 |
320 |
$10.15 |
$3,248.00 |
Instrument 5b: HPOG 2.0 National Evaluation letter and participant contact update form |
6,720 |
3,360 |
2 |
.1 |
672 |
$10.15 |
$6,820.80 |
Estimated Additional Annual Burden Previously Approved |
992 |
|
$10,068.80 |
||||
A12.3 Total Burden under OMB # 0970-0462
Exhibit A-10 shows the estimated respondent burden annualized over the remainder of the approval period is 9,143 hours. This represents the total burden remaining from the previously approved information collection, and the additional burden hours for previously approved information collection due to an increase in the number of respondents.
Exhibit A-10: Total Burden Remaining Under OMB #0970-0462
Instrument |
Annual Burden Hours |
Burden Remaining from Previously Approved Information Collection |
8,151 |
Additional Burden for Previously Approved Information Collection |
992 |
Total Annual Burden Hours |
9,143 |
Total Annual Cost
To compute the total estimated annual cost reported in Exhibit A-8 evaluators used the average wage for HPOG 1.0 participants employed at program intake ($10.64) and multiplied that by the proportion of those working at intake (0.37) for an average hourly total of $3.94. Evaluators believe the HPOG 1.0 data provide an accurate basis for estimating wages for HPOG 2.0 study participants for the previously approved information collection under PAGES. The baseline wage was appropriate for the original HPOG Next Generation submission as the PAGES system collects wage information at the time of enrollment. Since, this is a job training program the cost in the burden table in Supporting Statement A was revised to reflect the loaded federal minimum wage. The loaded federal minimum wage was used in the previously approved information collection requests for the HPOG 1.0 15- and 36-Month Follow-up Surveys and the PACE 15 and 36-Month Follow-up Surveys (OMB Nos. 0970-0394 and 0970-0397 respectively). For the cost to grantees and partner organizations data collection efforts, the total burden costs were multiplied by the average hourly wage, according to the Bureau of Labor Statistics, National Compensation Survey, 2010 ($28.29/hour). 14 The average hourly wage for the employer interviews is based on Bureau of Labor Statistics code 11-9111, Medical and Health Services Managers ($50.99). The evaluation team estimates that the annual costs for the remaining previously approved data collection is $112,022.35 annualized over the remainder of the approval period for consistency with other burden estimates under this clearance.
A13: Cost Burden to Respondents or Record Keepers
Not applicable. The proposed information collection activities do not place any new capital cost or cost of maintaining capital requirements on respondents.
A14: Estimate of Cost to the Federal Government
The evaluation team estimated the costs to complete the National and Tribal Evaluation based on their (1) experience with the first round HPOG 1.0 evaluations (OMB number 0970-0394); (2) experience with the Pathways for Advancing Careers and Education (PACE) study (OMB number 0970-0397), particularly for the Participant Interviews; and (3) experiences conducting longitudinal evaluations of hard-to-locate, low-income populations. The team carefully examined the different data collection approaches available to them and determined which options were most effective in their experiences on other similar studies. The cost estimates developed for the HPOG 2.0 National and Tribal Evaluations reflect our best estimates on the costs to conduct data collection with this population, and the most efficient methods for doing so.
The total cost for the data collection activities under the previously approved third revised submission were $5,477,757. Of that total, $3,630,491 was for the National Evaluation impact evaluation’s Intermediate Follow-up Survey and the phone-based Skills Assessment Pilot. The costs for the National Evaluation descriptive evaluation second-round telephone interviews and participant interviews were $961,184. The costs for the systems study interviews were $199,173 and the cost-benefit analysis study costs were $686,910.
The costs for all of the prior previously approved information collection requests under OMB Control Number 0970-0462 were $13,925,591. These costs included $8,473,750 for the National Evaluation impact evaluation’s Short-term Follow-up Survey, $1,788,164 for the National Evaluation descriptive evaluation, $418,236 for the increased burden for the previously approved instruments (see A12.2 and A15 for more detail), and $1,225,193 for the Tribal Evaluation, for a total of $11,905,343, plus $2,020,248 for the original submission.
Thus, the total costs to the Federal government for all information collections under this OMB control number (0970-04562) are $19,403,348.
Exhibit A-11below summarizes the costs to the Federal Government specifically associated with this previously approved information collection over the three year approval period. The table shows the three-year costs for four activity categories:
Ongoing PAGES data collection and reporting;
Design tasks (including instrument development, pretesting, OMB, and IRB activities);
Field Work (including impact study follow-up surveys, descriptive study interviews, participant interviews, focus groups, phone-based skills assessment pilot, and program cost surveys plus programming and training activities related to each); and
Analysis and Initial Reporting.
The table provides details on the number of total labor hours, the number of full-time equivalent staff persons they equate to, and loaded labor costs, plus operational expenses and other costs for each activity category.15 Operational expenses and other costs include such items as printing, equipment, overhead, shipping, travel, incentives, etc.
Exhibit A-11: Total Three Year Costs to the Federal Government under OMB #0970-0462 by Activity
Activity |
Labor Hours and Full-time Equivalents (FTEs) |
Labor Costs |
Operational Expenses and Other Costs |
Total Costs |
Ongoing PAGES data collection and reporting |
|
$191,922 |
$1,501,703
|
$1,693,625 |
Design |
|
$583,657 |
$380,774
|
$964,431 |
Field Work |
|
$5,174,490 |
$4,439,031
|
$9,613,521 |
Analysis and Initial Reporting |
|
$1,178,398 |
$703,795
|
$1,882,193 |
Total Costs Over Three Years |
|
$7,128,467 |
$7,025,303
|
$14,153,770 |
Annual Cost |
|
$2,376,155 |
$2,341,768
|
$4,717,923 |
A15: Change in Burden
As a result of the recent COVID-19 outbreak, most HPOG programs have had to adapt their programs in numerous ways—reduce class sizes, shift to virtual classes, suspend training temporarily, relax certification requirements or hands-on coursework, etc. The employment landscape has also changed for participants enrolling and participating in healthcare training programs during COVID-19, relative to those trained prior to the pandemic. In order to understand the effect of COVID-19 on the HPOG 2.0 Program, it will be important to understand the experiences of those enrolled after the onset of the pandemic compared to those who enrolled pre-pandemic. ACF is considering fielding a survey with a cohort of study participants enrolled during the COVID-19 Pandemic to better understand how COVID-19 effected the HPOG Program.
In the event ACF decides to fund the additional survey, it is important to notify study participants who enrolled after the onset of the pandemic period that they are part of a special cohort of study participants and may be selected to participate in a follow-up survey data collection effort. At this point, ACF seeks approval from OMB for a small burden increase for the Participant Contact Update Letter and Form (Instrument 5b) and the Welcome to the Study Packet and Contact Update Form (Instrument 5a), originally approved in June 2017, so they can begin outreach to study participants. .16 The increase in the number of respondents results in an increase of 1,984 hours annualized over the remainder of the approval period for consistency with other burden estimates under this clearance.
A16: Plan and Time Schedule for Information Collection, Tabulation and Publication
16.1 Analysis Plan
Exhibit A-12 summarizes the primary domains covered in the Short-Term and Intermediate Follow-up Survey instruments and provides a brief discussion of how they will be used.
Exhibit A-12: Domains for HPOG 2.0 Short-Term and Intermediate Follow-up Survey Instruments
Domain |
Notes |
Uses |
A. Training and employment history |
Dates of every school and job spell since randomization. Reasons for no school/job during ever (Short-term Follow-up Survey only). Careful probing for simultaneous study and work as well as multiple job holding. Dates of every school and job spell since randomization. |
|
B. School Experiences |
For each school spell:
|
|
C. Credential attainment and education/career goals |
Mostly about credentials, both those issued by schools and those issued by other authorities. |
|
D. Terms of employment and conditions at current/last job |
Occupation, scheduling, hourly wage rate, typical hours, benefits, other quality measures. |
|
E. Household composition |
Living arrangements, counts of adults and children, family formation, child bearing. |
|
F. Income and financial well-being |
Includes personal and household participation in government anti-poverty programs as well as income; includes questions on financial well-being and material hardship. |
|
G. 21st Century Skills |
Use of computer, literacy and numeracy skills at work and in everyday life; self-directed learning
|
|
H. COVID-19 |
Effects of COVID-19 on education and training, employment, and well-being |
|
I. Respondent and Secondary Contact Information |
Address, phone, and email information for the respondent and up to three people who know how to reach the respondent |
|
J. Basic Skills Assessment |
Basic skills (literacy and numeracy) assessment battery (Intermediate Survey Only) |
|
The HPOG 2.0 National Evaluation team will produce several reports using the data collected for the descriptive evaluation. The reports will include:
Descriptive Evaluation Reports. The Descriptive Evaluation will produce three reports: (1) an implementation study report; (2) a systems study report; and (3) a participant outcomes study report. The evaluation team will produce a series of briefs from the participant interview data. The evaluation will also use results from the implementation study to produce short case study reports on focus areas of specific interest to ACF.
Impact Evaluation Reports. Findings from the implementation study will inform the analysis in the evaluation’s impact evaluation reports. The evaluation is expected to produce reports on results based on the Short-term (15-month) and Intermediate (36-month) Follow-up Surveys and associated administrative data analysis. The team will also produce a longer-term (60-months post randomization) impact report using administrative data only.
Phone-based Skills Assessment Pilot. The evaluation team used separate three-parameter item response theory models to analyze test properties and respondent mathematical and vocabulary ability. The team developed static and dynamic subtests for each subject. For both, the questions were divided into “easy,” “medium,” and “hard” groups, to give a mix of the three types. Additionally, in both groups the team included questions with high relevancy scores and low guessing parameters. In an effort to keep the overall time of the module as short as possible, the evaluation team took the average item completion time into account for each question as well. Analysis of both the static and dynamic subtests for both subjects showed that the dynamic subtest option suited the study’s needs better than the static subtest for both subject areas of the final module.
The HPOG 2.0 Tribal Evaluation will use a systematic approach to analyze the data obtained through the interviews and focus groups conducted during and following annual site visits or remote data collection. The evaluation team will use NVivo software to store and analyze the large volume of data collected over the course of the evaluation. NVivo will be used to develop a coding scheme for analyzing these data. The coding scheme will be organized around evaluation topic areas derived from the evaluation questions. The coding scheme will be applied to all data and emergent key themes relating to evaluation topic areas will be identified.
The HPOG 2.0 Tribal Evaluation team will prepare a variety of reports, including site visit reports, practice briefs, and a final report.
Site Visit Reports. These reports will be developed after each annual site visit or remote data collection and summarize the findings from the interviews and focus groups.
Practice Briefs. Practice briefs will be shorter documents that highlight findings from the evaluation and share lessons learned.
Final Report. The final report will reflect the aggregated analysis of all qualitative and quantitative data collected throughout the evaluation
PAGES will automatically generate quantitative measures for the federally required semi-annual Program Performance Reports, which will include aggregated participant-level data, and will also store narrative-based grantee-level performance information. ACF will use these tables when preparing reports to Congress on the HPOG initiative. The PAGES team will produce a number of reports using data collected, including the six semi-annual PPRs and four annual reports.
The informed consent documents, whether signed or agreed to verbally, by the National and Tribal Evaluation study participants include permission for ACF and the evaluation team to archive the data. The specific plans for creating a public or restricted access dataset are still to be determined, but no data archiving will begin before the reports are published.
16.2 Time Schedule and Publications
The National Evaluation descriptive evaluation data collection began in July 2017, following OMB approval on the previous package. Contact updates for those participants in the National Evaluation impact evaluation sample began in November 2017 and will continue throughout the follow-up period. The National Evaluation impact study Short-term Follow-up Survey data collection began in October 2018 and will continue through the end of 2019. The Tribal Evaluation data collection began in October 2017. PAGES data collection will occur as individuals apply for the programs and enrollees receive training and services throughout the next three years of the grant period. Exhibit A-11 presents an overview of the project schedule for information collection.
Exhibit A-11: Project Schedule for Data Collection, Analysis, and Publication
Task |
Timing |
HPOG 2.0 National Evaluation: Descriptive Evaluation |
|
Descriptive evaluation data collection (includes, systems, and program implementation) |
June 2017-December 2019 |
National evaluation descriptive study site visits |
Fall 2018 |
Descriptive evaluation Analysis Plan |
Summer 2020 |
Systems Study Report |
Summer 2020 |
Implementation Report |
Fall 2020 |
Participant Interview Briefs |
Ongoing beginning Fall 2020 |
Outcomes Report |
Fall 2022 |
HPOG 2.0 National Evaluation: Impact Evaluation |
|
Contact update mailing for survey sample |
Quarterly beginning 3 months after random assignment (November 2017-Summer 2022) |
Short-term (15-Month) Follow-up Survey Data Collection |
October 2018-November 2019 (15 months after randomization for participants enrolled between March 2017 and February 2018) |
Intermediate (36-Month) Follow-up Survey Data Collection |
Beginning in September 2020 (approximately 36 months after randomization for selected participants enrolled between March 2017 and February 2018) |
Draft Short-term Impact Report to ACF |
Summer 2020 |
Data Archiving |
After publication of the Short-term Impact Report |
Draft Intermediate Impact Report to ACF |
Spring 2022 |
Data Archiving |
After publication of the Intermediate Impact Report |
Draft Longer-term Impact Report to ACF |
September 2024 |
HPOG 2.0 National Evaluation: Cost-benefit analysis study |
|
Program cost data collection |
Fall 2019 – Spring 2020 |
Draft Cost-Benefit report to ACF |
Spring 2022 |
HPOG 2.0 Tribal Evaluation |
|
Site visits or remote data collection for tribal grantees (1/year) |
Annually spring/summer of 2017, 2018, 2019, 2020 |
Conduct data analysis |
2017-2021 |
Develop Practice Briefs |
Annually September 2017-September 2021 |
Develop Final Report |
September 2021 |
HPOG 2.0 National and Tribal Evaluation Participant Accomplishment and Grantee Evaluation System (PAGES) |
|
PAGES grantee-level and ongoing participant-level data collection |
September 2015 – Winter 2022 |
6 Semi-annual PPRs |
September 2015 – September 2021 |
Two annual reports |
September 2015 – September 2020 |
A17: Reasons Not to Display OMB Expiration Date
All instruments created for the HPOG 2.0 National Evaluation and Tribal Evaluation will display the OMB approval number and the expiration date for OMB approval.
A18: Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 References to “counseling” refer to services such as tutoring, academic advising, financial aid advising, career counseling, job search or placement assistance, and case management.
2 Additional data sources will be used to answer the research questions, beyond those listed here. Only the data sources relevant to this submission are listed.
3 Please refer to Supporting Statements A and B, previously approved in June 2017, for more detail on the HPOG 2.0 Tribal Evaluation design.
4 Tribal Evaluation Workgroup. A Roadmap for Collaborative and Effective Evaluation in Tribal Communities. Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services. September 2013.
5 The HPOG 2.0 impact evaluation is a panel study. The three primary points of data collection are the previously approved Baseline Intake Form administered immediately prior to randomization, the previously approved Short-term Follow-up Survey, initiated 15 months after randomization, and the Intermediate Follow-up Survey, projected to begin 36 months after randomization (for which clearance is requested in this package). The impact evaluation will also include a longer-term follow-up evaluation, using administrative data only.
6 See Chapter 12 of Lynn (2009), in particular, section 12.5 that reviews the effects of incentives in several prominent panel studies.
7 See page 69, questions 75 and 76, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf
8 See updated Information Quality Act guidelines (https://www.whitehouse.gov/wp-content/uploads/2019/04/M-19-15.pdf)
9 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, 67 FR 8452 (Feb. 22, 2002), available at: https://www.federalregister.gov/documents/2002/02/22/R2-59/guidelines-for-ensuring-and-maximizing-the-quality-objectivity-utility-and-integrity-of-information
10 In accordance with HPOG funding requirements, the incentives can be redeemed only through vendors that do not sell alcohol, tobacco, firearms or other entertainment.
11 Kelly, B., Margolis, M., McCormack, L., LeBaron, P.A., Chowdhury, D. “What Affects People’s Willingness to Participate in Qualitative Research? An Experimental Comparison of Five Incentives.” Field Research. Volume: 29 issue: 4, page(s): 333-350
12 Two Tribal grantees will not collect social security numbers from some or all of their participants. A unique identifier will be assigned for these participants. Two versions of the Tribal informed consent forms were developed, one that includes social security numbers and one for grantees using unique identifiers.
13 The table reflects the increased burden for Instrument 5b, approved in July 2019.
14 Source: Bureau of Labor Statistics, National Compensation Survey, 2010: Combined average hourly wage across education training and library occupations and community and social services occupations.
15 The number of full-time equivalents (FTEs) is calculated by dividing the total number of labor hours by 2,080 workable labor hours per year.
16 It is unclear whether the grantees will maintain the same pace of enrollment during this extended period, thus the 6,400 participants is likely the upward limit.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Katheleen Linton |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |
© 2024 OMB.report | Privacy Policy