HPOG 2.0_Supporting Statement A_Revised_04152020_Temporary Approval of Verbal Consent
HPOG 2.0_Supporting Statement A_Revised_04152020_Temporary Approval of Verbal Consent.docx
OPRE Evaluation - National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants [descriptive evaluation, impact evaluation, cost-benefit analysis study, pilot study]
OMB: 0970-0462
Supporting Statement for OMB Clearance Request
Part A
National and Tribal Evaluation of the 2nd Generation of the Health Profession Opportunity Grants
0970-0462
Revision April 2019
Revised July 2019
Revised April 2020
Submitted by:
Office of
Planning,
Research & Evaluation
Administration for Children & Families
U.S. Department of
Health
and Human Services
Federal Project Officers:
Hilary Bruck
Nicole Constance
Amelia Popham
Table of Contents
A1: Necessity for the Data Collection 2
A2: Purpose of Survey and Data Collection Procedures 4
A3: Improved Information Technology to Reduce Burden 14
A4: Efforts to Identify Duplication 15
A5: Involvement of Small Organizations 17
A6: Consequences of Less Frequent Data Collection 17
A8: Federal Register Notice and Consultation 18
A9: Incentives for Respondents 19
A10: Privacy of Respondents 24
A12: Estimation of Information Collection Burden 28
A13: Cost Burden to Respondents or Record Keepers 32
A14: Estimate of Cost to the Federal Government 33
A16: Plan and Time Schedule for Information Collection, Tabulation and Publication 34
A17: Reasons Not to Display OMB Expiration Date 38
A18: Exceptions to Certification for Paperwork Reduction Act Submissions 38
Instruments
Previously Approved Instruments
Instrument 1: PAGES Grantee- and Participant-Level Data Items List
Instrument 2: HPOG 2.0 National Evaluation Screening Interview
Instrument 3: HPOG 2.0 National Evaluation first-round telephone interview protocol
Instrument 4: HPOG 2.0 National Evaluation in-person implementation interviews
Instrument 4A HPOG 2.0 National Evaluation In-Person Implementation Interview
Instrument 4B HPOG 2.0 National Evaluation In-Person Implementation Interviews Basic Skills Training
Instrument 4C HPOG 2.0 National Evaluation In-Person Implementation Interviews Career Pathways
Instrument 4D HPOG 2.0 National Evaluation In-Person Implementation Interviews Work-Readiness
Instrument 4E HPOG 2.0 National Evaluation In-Person Implementation Interviews Sustainability
Instrument 5: HPOG 2.0 National Evaluation welcome packet and participant contact update forms
Instrument 5a: HPOG 2.0 National Evaluation welcome packet and contact update form
Instrument 5b: HPOG 2.0 National Evaluation participant contact update letter and form
Instrument 6: HPOG 2.0 Tribal Evaluation grantee and partner administrative staff interviews
Instrument 7: HPOG 2.0 Tribal Evaluation program implementation staff interviews
Instrument 8: HPOG 2.0 Tribal Evaluation employer interviews
Instrument 9: HPOG 2.0 Tribal Evaluation program participant focus groups
Instrument 10: HPOG 2.0 Tribal Evaluation program participant completer interviews
Instrument 11: HPOG 2.0 Tribal Evaluation program participant non-completer interviews
Instrument 12: HPOG 2.0 National Evaluation Short-term Follow-up Survey
Instrument 13:HPOG 2.0 Screening Interview Second Round
Instrument 14:HPOG 2.0 Second Round Telephone Interview Guide
Instrument 15:HPOG 2.0 Program Operator Interview Guide for Systems Study
Instrument 16:HPOG 2.0 Partner Interview Guide for Systems Study
Instrument 17:HPOG 2.0 Participant In-depth Interview Guide
Instrument 18:HPOG 2.0 Intermediate Follow-up Survey
Instrument 19:HPOG 2.0 Phone-based Skills Assessment Pilot Study Instrument
Instrument 20:HPOG 2.0 Program Cost Survey
Attachments
New Attachments Included in This Change Request
Attachment B: New Verbal Informed Consent Forms, Updated for Verbal Consent
Attachment B: National Evaluation Informed Consent form C (Lottery Required)_Verbal
Attachment B: National Evaluation Informed Consent form D (Lottery Not Required)_Verbal
Attachment B2: Tribal Evaluation Informed Consent form C (SSNs)_Verbal
Attachment B3: Tribal Evaluation Informed Consent form D (Unique identifiers)_Verbal
Previously Approved Attachments
Attachment A: References
Attachment B: Previously Approved Informed Consent Forms
Attachment B: National Evaluation Informed Consent form A (Lottery Required)
Attachment B: National Evaluation Informed Consent form B (Lottery Not Required)
Attachment B2: Tribal Evaluation Informed Consent form A (SSNs)
Attachment B3: Tribal Evaluation Informed Consent form B (Unique identifiers)
Attachment C: 60-Day Federal Register Notice
Attachment D: Previously Approved Sources and Justification for PAGES Grantee- and Participant-Level Data Items
Attachment E: Previously Approved Final Updated Attachment E PPR Data List and Mockup
Attachment F: First Round of HPOG Grantees Research Portfolio
Attachment G: Previously Approved Participant Contact Information Update Letter and Form (Obsolete, replaced by Instrument 5a and 5b)
Attachment H: HPOG Logic Model
Attachment I: Previously Approved Focus Group Participant Consent Form
Attachment J: Previously Approved Interview Verbal Informed Consent Form
Attachment K: HPOG 2.0 National Evaluation Short-term Follow-up Survey Advance Letter
Attachment L: HPOG 2.0 National Evaluation Short-term Follow-up Survey Sources
Attachment M: HPOG 2.0 National Evaluation Short-term Follow-up Survey Trying to Reach You Flyer
Attachment N: HPOG 2.0 National Evaluation Short-term Follow-up Survey Email Reminder
Attachment O: Research Questions for Previously Approved Data Collection Efforts (National Evaluation and Tribal Evaluation)
Attachment P: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Advance Letter
Attachment Q: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Sources
Attachment R: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Trying to Reach You Flyer
Attachment S: HPOG 2.0 National Evaluation Intermediate Follow-up Survey Email Reminder
Attachment T: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot flyer
Attachment U: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot grantee letter
Attachment V: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot participant letter
Attachment W: HPOG 2.0 National Evaluation phone-based Skills Assessment Pilot recruitment script
Attachment X: Complete list of previously approved data collection instruments
Attachment Y: 60-day Federal Register Notice
Attachment Z: Participant Interview Recruitment Materials
Overview The
revisions in this Supporting Statement reflect changes requested as
part of a non-substantive change request, submitted to OMB in April
2020. As described
in an accompanying memo, ACF is seeking approval for a new set of
informed consent documents for the National and Tribal Evaluations
of the 2nd
Generation of the Health Profession Opportunity Grants (HPOG 2.0).
These revised consent forms allow for verbal consent rather than
written consent, a temporary change necessitated by the COVID-19
outbreak. COVID-19 has prompted a temporary pause in in-person
intake and enrollment into most HPOG programs and any in-person
data collection to protect both study participants and program
staff.
The
majority of the changes to the consent process and forms that are
the subject of the April 2020 non-substantive change request memo
are found in sections A1 and A2, as well as Supporting Statement B,
Sections B1 and B2. Minor updates are included throughout both
Supporting Statements as needed.
Part A: Justification
The National and Tribal Evaluations of the Health Profession Opportunity Grants (HPOG) program, known as the HPOG 2.0 Evaluation is funded by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS).
The HPOG 2.0 Evaluation includes evaluations of tribal grantees (Tribal Evaluation) and non-tribal grantees (National Evaluation).In July 2019, OMB approved ACF’s request for a third revision to the data collection plans for the HPOG 2.0 Evaluation, previously approved under OMB Control No. 0970-0462.
Exhibit A-1 first reviews the content of the original request plus the three approved revisions. As a result of the recent COVID-19 outbreak, most HPOG programs have changed to operate their programs virtually, including asking staff to work remotely and offering participant trainings online. These restrictions have brought the face-to-face intake and enrollment sessions to a sudden halt in most locations. However, grantees are eager to continue enrollment in the interest of serving the healthcare workforce needs in their communities. This request is for a non-substantive change to the National and Tribal Evaluation informed consent documents and procedures, updating them for verbal consent. The current consent forms (Attachments B, B2, and B3) were originally approved in August 2015, revised in January 2016, and renewed most recently in July 2019. Those previously approved forms require written signatures, which is extremely difficult to obtain when face-to-face intake and enrollment sessions are halted. ACF is requesting approval for a new, alternative set of informed consent forms. These new consent forms will temporarily allow participants to verbally consent into the study until grantees can return to their worksites and resume normal program operations (See verbal versions of Attachments B, B2, and B3). Once normal operations resume, grantees will resume using the previously approved consent forms and requiring written consent to join the study. An overview of these non-substantive changes can be found in the supplementary document OMB#_0970-0462_Non-substantive Change Request _Temporary Use of Verbal Consent Forms_April 2020.docx.
Exhibit A-1: Clearance Requests and Instruments for HPOG 2.0 (OMB Control No. 0970-0462)
Request |
Instrument(s) |
Request Date |
Approval Date |
Link to Supporting Statement |
Original |
Participant Accomplishment and Grant Evaluation System (PAGES) (Instrument #1) |
5/13/15 |
8/6/15 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201505-0970-002 |
1st Rev. |
Various baseline, process and contact update forms (Instruments #2-5b for the National Evaluation; #6-11 for the Tribal Evaluation) |
10/26/16 |
6/27/17 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201610-0970-012 |
2nd Rev. |
National Evaluation Short-term Follow-Up Survey (Instrument #12) |
2/5/18 |
6/8/18 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201802-0970-001 |
3rd Rev. |
Additional National Evaluation data collection tools: Descriptive evaluation protocols (Instruments #13-17); Intermediate Follow-up Survey (Instrument #18); Phone-based Skills Assessment Pilot (Instrument #19); and Program Cost Survey (Instrument #20). |
4/23/2019 |
7/24/19 |
https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201904-0970-006
|
Non-substantive Change Request-Post July 2019 Approval of Revision #3 (this submission) |
Non-substantive Change Request for a new set of alternative consent forms and procedures allowing for verbal consent temporarily during COVID-19
|
April 2020 |
|
|
A1: Necessity for the Data Collection
A1.1 Study Background
The HPOG Program, established by the Patient Protection and Affordable Care Act of 2010 (ACA), funds training in high-demand healthcare professions, targeted to TANF recipients and other low-income individuals. The HPOG Program is administered by the ACF Office of Family Assistance (OFA). The first round of HPOG grants was awarded in 2010. In September 2015, OFA awarded a second round of HPOG grants—valued at approximately $72 million—to 32 organizations located across 21 states. Grantees include six community based organizations, four state government entities, seven local workforce development agencies, ten institutions of higher education, two tribal colleges, one tribal human service agency, one tribe, and one Indian Health Board. Those 32 grantees oversee 43 individual HPOG programs—five tribal programs and 38 non-tribal programs. All grantees are participating in this federal evaluation.
HPOG programs: (1) target skills and competencies demanded by the healthcare industry; (2) support career pathways, such as an articulated career ladder; (3) result in an employer- or industry-recognized credential (which can include a license, third-party certification, postsecondary educational certificate or degree, as well as a Registered Apprenticeship certificate); and (4) combine supportive services with education and training services to help participants overcome barriers to employment, as necessary.
HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. See A1.2 for additional information. The federal evaluation activities are intended to expand the career pathways evidence base and to build on what has been learned to date about how to design and implement successful career pathways programs for low-income and low-skilled individuals, and improve the outcomes of individuals who participate in these programs. The federal evaluation for the non-tribal HPOG 2.0 grantees involves random assignment of individual participants. Tribal grantees are participating in a coordinated evaluation that does not involve random assignment.
Abt Associates is the prime contractor and the lead for the HPOG 2.0 National Evaluation. Abt and the Urban Institute led the design of the Participant Accomplishment and Grant Evaluation System (PAGES), which collects and stores uniform data needed for performance management and the federal evaluations, incorporating the required semi-annual grantee performance reports to ACF (Attachment E). Both organizations are overseeing PAGES data collection. Partners MEF Associates, Insight Policy Research and the Urban Institute are assisting with the site monitoring, descriptive evaluation, and cost-benefit analysis. NORC at the University of Chicago is leading the HPOG 2.0 Tribal Evaluation under subcontract to Abt Associates.
A1.2 Legal or Administrative Requirements that Necessitate the Collection
The HPOG Program is authorized under section 2008(a)(1) of the Social Security Act (42 U.S.C. § 1397g(a)(1)), and extended by the “Coronavirus Aid, Relief, and Economic Security Act” or the “CARES Act of 2020, Pub. L. 116-136 through the first two months of fiscal year 2021. H.R. 3590, the ACA requires an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(C)). The Act calls for evaluation activities to assess the success of HPOG in “creating opportunities for developing and sustaining, particularly with respect to low-income individuals and other entry-level workers, a health professions workforce that has accessible entry points, that meets high standards for education, training, certification, and professional development, and that provides increased wages and affordable benefits, including healthcare coverage, that are responsive to the workforce’s needs” (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)).
There were two Funding Opportunity Announcements (FOAs) for the second round of HPOG grants—one for non-tribal grantees (HHS-2015-ACF-OFA-FX-0951) and one for Tribal grantees (HHS-2015-ACF-OFA-FY-0952). Both FOAs required all HPOG 2.0 grantees to participate in a federal evaluation and to follow all evaluation protocols established by ACF or its designated contractors. Participating in the federal evaluations includes the use of the PAGES data system to collect uniform data elements and, for non-tribal grantees, the facilitation of random assignment.
HPOG’s authorizing legislation calls for a comprehensive evaluation of the funded demonstration projects. Accordingly, ACF is rigorously evaluating the effectiveness of funded HPOG 2.0 programs. Data collected under PAGES is used to: automatically generate the federally required semi-annual program performance reports; inform ACF reports to Congress; monitor and manage the performance of the grant-funded projects; inform the HPOG 2.0 National Evaluation impact, outcomes, and implementation studies and HPOG 2.0 Tribal Evaluation; and inform other future research and evaluation efforts. Data collected through the previously approved instruments—t—support the HPOG 2.0 National and Tribal Evaluations.
A2: Purpose of Survey and Data Collection Procedures
A2.1 Overview of Purpose and Approach
The data collection instruments described in this section were previously approved in July 2019. This submission seeks approval of non-substantive changes to the previously approved informed consent forms and procedures.
HPOG 2.0 Informed Consent-National and Tribal Evaluations
ACF is seeking approval of a non-substantive change to add a new set of informed consent forms to the clearance package for OMB Control #0970-0462. These alternative forms will allow grantee staff to change the way informed consent is obtained in instances where face-to-face enrollment is not feasible due to issues related to the COVID-19 pandemic.
When face-to-face enrollment is once again feasible, the consent and intake data collection procedures will return to the procedures as previously approved:
All non-Tribal grantees are required to participate in the impact evaluation. During the intake processes for grantees in the study, staff first will describe the impact study and administer the random assignment informed consent form (see Attachment B). All eligible applicants for non-Tribal grantee programs must sign the paper informed consent form during the intake period for the study in order to be part of the study.1
Flexibility in how consent is obtained will ensure that participant enrollment can continue and that grantees are able to serve participants and sustain program operations during this time. Verbal consent is vital to maintaining the rigor of the evaluation in order to ensure that participants enrolled during this period can be included in the impact and descriptive evaluations of the National Evaluation and in the Tribal Evaluation. These participants are not part of any survey data collection efforts at this time. In particular, the administrative data from the National Directory of New Hires (NDNH) is crucial for measuring employment and earnings outcomes for this group of participants.2 The revised procedures for obtaining verbal consent are described in Supporting Statement B, Section B2.
HPOG 2.0 National Evaluation
The National Evaluation involves random assignment of individual participants. As stated in the FOA, the non-tribal HPOG 2.0 grantees are required to abide by random assignment procedures and facilitate the random assignment process for individuals by entering eligible HPOG program applicants into a lottery to determine if they will be invited to participate in the program. Applicants who are not invited to participate serve as a control group in the evaluation. The control group members do not receive HPOG program services, but may enroll in any other program or service for which they are eligible. Individuals must complete the application process prior to random assignment; only individuals who have been deemed eligible for program participation may be entered into the lottery.
For the National Evaluation, the third revised ICR submission, approved in July 2019, covered instruments for the descriptive evaluation, the impact evaluation, and the cost-benefit analysis study. The eight instruments in the third revised submission, approved in July 2019, were:
Five instruments for the descriptive evaluation (Instruments 13-17)
Screening Interview Guide, Second-round Interviews,
Second-round Telephone Interview Guide,
Program Operator Interview Guide for systems study,
Partner Interview Guide for systems study, and
Participant In-depth Interview Guide;
Two instruments for the impact evaluation (Instruments 18 and 19)
Intermediate Follow-up Survey,
Phone-based Skills Assessment Pilot; and
One instrument for the cost-benefit analysis study (Instrument 20)
Program cost survey.
A2.2 Research Questions
This non-substantive change request does not require any changes to the research questions included in prior versions of the Supporting Statement, nor to this section of the Supporting Statement. The third revised request for clearance, approved in July 2019, covered the research questions applicable to the National Evaluation impact evaluation—to be addressed by the Intermediate Follow-up Survey and the phone-based Skills Assessment Pilot; as well as the additional data to be collected in support of the descriptive evaluation and the cost-benefit analysis study.
The research questions for the National Evaluation descriptive evaluation and the Tribal Evaluation were summarized in a previously approved request for clearance, along with their respective data collection protocols (OMB Control Number 0970-0462, approved June 2017). The research questions from that prior submission are included in Attachment O.
Exhibit A-2 provides a schematic for the theory of action for the HPOG 2.0 National Evaluation’s impact evaluation. The top row of the exhibit represents the experiences of applicants randomized to the treatment group—that is, those offered a “slot” in an HPOG 2.0 program, where “slot” means the package of training and associated support services offered by the program, whether or not the individual uses the components of that package. Conversely, the bottom row represents the experiences of those in the control group, who are not offered an HPOG 2.0 slot.
Exhibit A-2: Schematic and Theory of Action of the HPOG 2.0 Impact Evaluation
From left to right in the top row of the exhibit, an applicant randomly assigned to the treatment group is offered an HPOG 2.0 slot and thereby gets access to the training and associated support services from the HPOG 2.0 program and, potentially, from other sources. (Nevertheless, but not explicitly shown in the exhibit, not everyone offered access to HPOG services will use everything—or even anything—offered.) The hypothesis to be tested is that HPOG’s package of training and associated support services leads to impacts on educational and occupational credentials, employment, working conditions (hours, hourly wage, shift work, benefits), and earnings. Impacts on public assistance receipt (TANF, SNAP, Medicaid, and unemployment insurance) and broader aspects of well-being (food security, housing stability, and marital status) may also emerge.
In contrast, those randomly assigned to the control group (the bottom row of the exhibit) are not offered access to the HPOG 2.0 program, but may obtain training and other support services from other sources. The same set of outcomes emerges, though possibly at different levels and with different timing: education and credentials, employment and earnings, public assistance, and overall well-being.
Though not everyone in the treatment group gets training and many in the control group do get training, the two contrasting flows in Exhibit A-2 represent the very contrast relevant to future policy decisions on funding HPOG-like services. Random assignment creates a treatment group and a control group that differ only by the offer of HPOG 2.0 and chance. Because the two groups are otherwise statistically equivalent, comparisons of outcomes between them provides an unambiguous estimate of the impact of HPOG 2.0; “impact,” refers to outcomes for those offered HPOG in a world with the program relative to what outcomes for those same individuals would have been had HPOG not existed.
The National Evaluation considers all aspects of the theory of action shown in Exhibit A-2:
Addressing implementation research questions, the descriptive evaluation describes the HPOG 2.0 program as implemented.
Addressing service contrast research questions, the service contrast analysis estimates the impact of the offer of HPOG 2.0 on services received—training and other support services.3
Addressing impact research questions, the impact analysis estimates the impact of the offer of HPOG 2.0 on outcomes of interest—including educational programs completed, credentials received, employment, earnings, and participation in public assistance programs.
Assessing the costs of implementing the program relative to the benefit to participants and grantees is the primary goal of the cost-benefit analysis.
Exhibit A-3 provides the research questions that pertained to the data collection activities included in the third revised request for clearance, approved in July 2019.
Exhibit A-3: Research Questions Relevant to the Third Revision Approved in July 2019
Evaluation Component |
Data Source4 |
Research Questions |
Descriptive Evaluation |
Second-round telephone interviews |
|
Participant In-depth interviews |
|
|
Systems study telephone interviews with program operators and partner organizations |
|
|
Impact Evaluation |
Intermediate Follow-up survey |
Research questions on the impact of being offered an HPOG 2.0 slot:
The research questions above are framed as the impact of being offered a slot. If sufficient resources are available, the evaluators will address an additional research question:
|
Skills Assessment Pilot |
The Skills Assessment Pilot is intended to determine experimentally if programs such as HPOG 2.0 improve the kinds of basic skills deemed most relevant to qualifying students to participate in education and training programs designed to prepare them for healthcare occupations. The pilot will help to identify a parsimonious subset of items that are relevant to success in healthcare training and that achieve the right mix of level of difficulty, as determined though item response modeling (IRT). It will also determine if these skills can be assessed over the telephone. Once identified, this subset of items will be incorporated into the Intermediate Follow-up Survey as a separate module at the end, in order to address this research question:
|
|
Cost-benefit analysis study |
Program cost survey |
|
A2.3 Study Design
This section provides an overview of the design of the HPOG 2.0 National Evaluation. For an overview of the Tribal Evaluation study design, please see the first revision under OMB Control No. 0970-0462 approved in June 2017. For more detail on the previously approved PAGES system, please see the original submission under OMB Control No. 0970-0462 (approved in August 2015). PAGES was designed to meet the performance data needs of the grantees and of OFA to monitor the grantee performance and prepare the report to Congress on the grants. PAGES supports the National and Tribal Evaluations, as well as other future research and evaluation efforts sponsored by ACF.
HPOG 2.0 National Evaluation Study Design
The HPOG 2.0 National Evaluation is guided by the career pathways framework, as shown in the HPOG logic model (Attachment H). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012). These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports and employer connections.
The design for the HPOG 2.0 National Evaluation features a
descriptive evaluation (including an implementation study, a systems study, and an outcome study);
an impact evaluation (using a classic experimental design to measure and analyze key participant outcomes including completion of education and training, receipt of certificates and/or degrees, earnings, and employment in a healthcare career); and
a cost-benefit analysis study.
Exhibit A-4 provides a visual description of the major components and sub-components of the HPOG 2.0 National Evaluation.
Exhibit A-4: Components of the HPOG 2.0 National Evaluation
Briefly, and as discussed above, the impact evaluation design includes randomizing program-eligible participants to treatment and control status in all non-tribal sites. Follow-up to answer the research questions will involve both queries of administrative data systems and participant surveys. The surveys are described in more detail in Section A4 below. The research team will match participant data collected through the impact evaluation for both the treatment and control groups to long-term employment and earnings data from ACF’s National Directory of New Hires (NDNH) and to school enrollment data from the National Student Clearinghouse (NSC). An agreement with the Office of Child Support Enforcement (OCSE) to use the NDNH data was signed in February 2018. Negotiations to use the NSC data are underway.
A2.4 Universe of Data Collection Efforts
To address these research questions, the HPOG 2.0 National Evaluation will use a number of data collection instruments. Exhibit A-5 describes the target respondents, content, and reason for collection (i.e., which analyses will use the information) for each of the data collection activities submitted with the third revised submission request and approved in July 2019. All other support materials (for example the survey advance letter or the survey flyers) for the new instruments are provided in Attachments P, R, S, T, U, V, W, and Z.
For a list of study instruments approved by OMB in prior information collection requests, see Attachment X.
Exhibit A-5: HPOG 2.0 National Evaluation Instrument Overview-with Completion Status
Data Collection Activity |
Data Collection Instrument(s) |
Respondents, Content, and Reason for Collection |
Descriptive Evaluation Second-Round Telephone Interviews (COMPLETE) |
Screening Interview to identify respondents for the HPOG 2.0 National Evaluation second-round telephone interviews (Instrument 13)
HPOG 2.0 National Evaluation second-round telephone interview guide (Instrument 14) |
Respondents: Program staff, managers, partners, and stakeholders at the 38 programs administered by the 27 non-tribal grantees. The telephone interviews took place in Fall 2019.
Content:
Reason: The screening interview was conducted over the phone with HPOG program managers to identify appropriate respondents for the second-round telephone interviews. The second-round telephone interviews built on the first round of interviews (previously approved in June 2017 under OMB Control No. 0970-0462). These interviews collected information about the HPOG program context and about program administration, activities and services, partner and stakeholder roles and networks, and respondent perceptions of the program’s strengths. The second round placed additional focus on implementation and performance successes and challenges; documentation of changes since the start of the grant; and information on how grantees spend their grant funds. Information gathered during interviews inform the descriptive evaluation’s implementation study. These data are not available through any current sources. |
Descriptive Evaluation Systems Study Interviews (COMPLETE) |
HPOG 2.0 National Evaluation systems study interview guides (Instruments 15 and 16) |
Respondents: Program staff and partners at a subset of purposively selected non-tribal HPOG programs (12-16 of the 38 programs). The telephone interviews occurred in fall 2019.
Content:
Reason: Systems study interviews explored whether and how the HPOG 2.0 grants supported systems activities for providing healthcare training opportunities to low-income adults. The interviews were the key data source for the systems study, which builds on findings about the program operations and partnerships described by the Implementation Study to learn from HPOG programs and the systems in which they operate. These data are not available through any current sources. |
Descriptive Evaluation Participant Interviews (COMPLETE) |
HPOG 2.0 participant interview guide (Instrument 17) |
Respondents: Expected sample of 140 treatment group members, across 14 HPOG programs. The in-person interviews we conducted in fall 2019 through early winter 2020.
Content:
Reason: The goal of these in-depth qualitative interviews was to gain insights into the motivations, decision making, expectations, and experiences of HPOG 2.0 Program participants. These data are not available through any current sources. Many of the questions asked in the interviews were based on questions approved for ACF’s Pathways for Advancing Careers and Education participant interviews (OMB #: 0970-0397) to offer continuity in the information collected across ACF’s career pathways research portfolio. |
Impact Evaluation Participant Follow-up Survey 36 months after randomization |
Intermediate Follow-up Survey for the HPOG 2.0 National Evaluation (Instrument 18) |
Respondents: Overall expected sample of 5,000 (selected participants randomized between March 2017 and February 2018). Data collection will begin in September 2020 (by telephone and then in-person for those respondents who cannot be reached by telephone).
Content:
Reason: The Intermediate Follow-up Survey will collect information on events that have occurred since the Short-term Follow-up Survey (approximately 15-months post-randomization) in many areas—particularly the receipt of training and related supports, and receipt of credentials. This survey information will provide outcomes for the impact analysis. These data are not available through any current sources. Many of the questions to be asked in this survey were approved for the Short-term Follow-up Survey for the HPOG 2.0 National Evaluation (OMB Control No. 0970-0462), along with other studies in ACF’s Career Pathways portfolio, specifically the Pathways for Advancing Careers and Education 15-, 36-, and 72-month follow-up surveys (OMB #: 0970-0397); and the impact study of the first round of HPOG 15-, 36-, and 72-month surveys (OMB #: 0970-0394). (A summary of the 36-month survey item sources is provided in Attachment Q.) Note: The draft Intermediate Follow-up Survey includes all of the items from the pilot assessment (see next line). Based on the findings from the pilot we will drop questions that do not prove successful in the pilot and will submit the final Intermediate Follow-up Survey to OMB as a non-substantive change request. |
Impact Evaluation phone-based Skills Assessment Pilot (COMPLETE) |
Questionnaire for a phone-based Skills Assessment Pilot (Instrument 19) |
Respondents: Expected sample of 500 HPOG 2.0 participants. Non-tribal grantees will be asked to help identify potential volunteers to participate. The phone-based skills assessments were completed in late fall 2019.
Content:
Reason: The logic models for career pathways programs typically include a path toward increasing earnings. Accordingly, it is important to determine experimentally if programs such as HPOG 2.0 improve the kinds of basic skills deemed most relevant to qualifying students to participate in education and training programs designed to prepare them for healthcare occupations. Since telephone interviews with study subjects almost always play an important role in experimental evaluations of these programs, it would be useful to have a module that measures program participants’ level of these basic skills over the phone. The mode and burden of collecting this type of basic skills information with other methods—in-person or online—can be quite expensive and burdensome on participants, making these methods infeasible for a large-scale evaluation like HPOG 2.0 National Evaluation. The phone-based Skills Assessment Pilot administered a series of 45 literacy and numeracy based assessment questions in order to identify a shorter subset —12 vocabulary-based and 12 math-based questions—for inclusion in the Intermediate Follow-up Survey. The subset included in the Intermediate Follow-up Survey will be used to assess respondents’ literacy and numeracy experimentally. Respondents will only be asked up to 16 of the 24 questions included in the survey, eight in each category. The items for inclusion in the revised Instrument 18 are the subject of a forthcoming non-substantive change request. These data are not available through any current sources. |
Cost-benefit analysis study Program cost data collection (COMPLETE) |
HPOG 2.0 Program Cost Survey (Instrument 20) |
Respondents: Staff at the 38 non-tribal HPOG programs. The cost surveys were completed between fall 2019 and spring 2020.
Content:
Reason: The purpose of the cost survey was to gather information on HPOG 2.0 program costs, including expenditures on staff, overhead, academic supports, employment supports, and other supports. The Cost-benefit analysis study will use results of this survey to estimate costs per participant for academic and employment supports, assistance with transportation and training-related materials (including school fees, supplies, uniforms, testing fees, equipment and tools), as well as to measure the costs of program administration. These data are not available through any current sources. |
Other extant data sources will be used for the HPOG 2.0 National and Tribal Evaluation. These include the following:
National Directory of New Hires (NDNH). These data will provide information on employment and earnings of HPOG participants.
National Student Clearinghouse (NSC). These data will provide information on student enrollment in credit-bearing courses (and some enrollment in non-credit bearing courses) and receipt of post-secondary degrees.
HPOG program management information, including initial applications and ongoing management reports, which will provide supplemental information in tracking the evaluation of the grant, and information on the local healthcare labor market and needs for occupational training.
Government sources of labor market data, from the U.S. Census Bureau and Bureau of Labor Statistics (BLS), such as County Business Patterns, Local Area Unemployment Statistics (LAUS), and Quarterly Workforce Indicators (QWI), which will provide a picture of the local labor market.
A3: Improved Information Technology to Reduce Burden
The HPOG 2.0 National Evaluation and HPOG 2.0 Tribal Evaluations will generate a substantial amount of data using a combination of data collection methods. The evaluation team designs each data collection protocol to limit the reporting burden for respondents. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. As described in the originally approved supporting statement (approved in August 2015, with revisions in January and July 2016, June 2017, and June 2018), participant- and grantee-level data will be collected through PAGES, a cloud-based data system. The evaluation teams will use the quantitative data collected through PAGES to reduce respondent burden wherever possible. The team will rely on data collected during the first-round telephone interviews (see Section A4 for more detail) to pre-populate the second-round interviews where possible. The team will also rely on administrative data—such as NDNH—to capture employment and wage data. This removes the burden of collecting this information from participants during the follow-up survey. Any requests for program documentation will be collected electronically as well.
The HPOG 2.0 National Evaluation impact evaluation will offer study participants the option to update their contact information online, by mail, or by telephone. The follow-up surveys will be administered using computer assisted personal interviewing (CAPI) technology for all interviews. CAPI technology reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns are properly implemented, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who did not participate in postsecondary training will be routed past questions only relevant to those who did. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.
The HPOG 2.0 National Evaluation impact evaluation will use improved information technology to pilot a new way to assess basic skills. Numerous tools to measure basic skills already exist, but most require in-person or online data collection capabilities. In-person assessments are often too costly for most large-scale studies, and they put more burden on respondents. Online assessments are not always feasible for large-scale studies with populations similar to the HPOG 2.0 National Evaluation as some participants may not have computer or internet access. The phone-based Skills Assessment Pilot will allow the evaluation team to assess the feasibility of conducting skills assessments by telephone. If successful, a phone-based method to assess basic skills offers an alternative that reduces the costs to the Federal Government and reduces the burden on the respondents.
A4: Efforts to Identify Duplication
Abt and their partners are also conducting several other evaluations on behalf of ACF as part of the HPOG research portfolio on the first round of HPOG grantees, for which there are several data collections already approved by OMB (see Attachment F for further details). ACF and its contractors are engaged in many efforts to coordinate these evaluation activities and avoid duplication of work. The HPOG 2.0 Evaluation team has used the extensive knowledge generated to date from the research activities on the first round of HPOG and ACF’s Pathways for Advancing Careers and Education (PACE) programs (OMB control numbers 0970-0394 and 0970-0397, respectively) to inform the proposed new data collection efforts for the second round of HPOG grantees. This section summarizes those efforts for the National Evaluation impact evaluation, descriptive evaluation, and cost-benefit analysis study.
Impact Evaluation. The purpose of the HPOG 2.0 National Evaluation’s Intermediate Follow-up Survey is to obtain current information on the status and wellbeing of individuals in the HPOG 2.0 National Evaluation study sample members 36 months after randomization. Information about these respondents' educational achievement, economic well-being, job skills development and progression, and overall well-being are not consistently available through any other source, nor is information about family composition, student debt, or 21st century skills. The evaluation will utilize administrative data (e.g., wage records) in conjunction with survey data to avoid duplication of reporting.
The research team will also avoid duplication in this study by use of a study-specific database, maintained by Abt, which links all the data collected at baseline with subsequent information gathered from future surveys and administrative sources. This eliminates the need to ask about personal characteristics or background factors for known household members on follow-up surveys.
The phone-based Skills Assessment Pilot survey collected data from HPOG participants that were not available through any other source. Although HPOG 2.0 grantees may conduct their own basic skills assessments at intake, these assessments cannot substitute for this information collection. First, the assessment protocols that the programs use are quite diverse across grantees. Second, these assessments are generally only administered prior to randomization and thus are not available to examine effects of the HPOG Program. Finally, the exact scores of assessments done at enrollment are mostly missing in PAGES because they are not required fields. To address this issue, our proposed approach has the potential to identify a short subset of questions that can be incorporated into the Intermediate Follow-up Survey. This subset of questions will help to assess basic skills of HPOG participants and thereby be informative about the adequacy of preparation of HPOG 2.0 participants for the early phases of their healthcare career.
The pilot also tested the feasibility of telephone administration for this type of assessment. Several national and international surveys have been developed to assess adult numeracy and literacy, but almost all of these rely on face-to-face interviewing or online administration. Since the Intermediate Follow-up Survey will be administered over the telephone and in-person, the phone-based Skills Assessment Pilot will help to ensure that a trained interviewer can administer the assessment questions by telephone, making it appealing for HPOG 2.0 and potentially other OPRE evaluations as well. The recommendations from the now completed pilot study analysis will be reflected in a forthcoming non-substantive change request.
Descriptive evaluation. The purposes of the HPOG 2.0 National Evaluation second-round telephone interviews, the telephone interviews for the systems study, and the participant interviews are to obtain information about HPOG grantee programs, partners, and participants that is not available through any other source. Wherever possible, the research team will use existing sources of information—including PAGES, available information from the site-specific evaluation design and other existing site-specific materials developed earlier by the National Evaluation team—to collect key information prior to conducting the telephone interviews. These existing data sources and information available from each include the following:
Evaluation Design and Other Site-Specific Materials
Target population and recruitment strategies
Eligibility criteria and application processes
Control group services and conditions
PAGES and Program Performance Progress Reports (PPRs)
Available training courses and support services
Participant characteristics
Participant take-up of training courses and support services
Participant education and employment outcomes
Site Team Monitoring Reports
Documented program changes
Grantee applications
Organization type
Cost-benefit analysis study. The purpose of the cost data collection was to obtain information about HPOG grantees program costs that is not available through any other source. The data collected will complement other data sources, not duplicate or replace them. The research team will use other existing sources of information to estimate costs, including the following:
Delta Cost Project Database (DCPD): A publicly available longitudinal database derived from the U.S. Department of Education’s Integrated Postsecondary Education Data System (IPEDS). The database translates IPEDS information into analytic formats for analysis of revenues and expenditures in postsecondary education.
Follow-up Surveys of Study Participants: The Short-term (15-month) Follow-up Survey approved in June 2018 and the Intermediate (36-month) Follow-up Survey included in this submission.
PAGES: Data from programs on participant receipt of training and services funded by their grants.
A5: Involvement of Small Organizations
The National Evaluation and Tribal Evaluation will have minimal impact on small organizations. The primary organizations involved in this study are tribal and community colleges, workforce development agencies, tribal organizations, and community-based organizations that operate occupational training programs. The funding announcement informed all grantees of the federal evaluation and reporting requirements, and adequate resources have been provided to coordinate the data collection and reporting. There should be no adverse impact for any grantees participating in the study.
For the descriptive study, small business professionals will only be interviewed if they are HPOG grantee partners or employers of HPOG program graduates. Interviewers will make every effort to complete these interviews as efficiently as possible.
There is no small business involvement in the other data collection efforts included in this submission.
A6: Consequences of Less Frequent Data Collection
This section summarizes the consequences of less frequent data collection for the HPOG 2.0 National Evaluation impact evaluation, descriptive evaluation, and cost-benefit analysis study.
Impact evaluation. For the HPOG 2.0 National Evaluation impact evaluation, the evaluation team plans only two rounds of substantive data collection with individual participants. The Short-term Survey started at 15 months following randomization, and the Intermediate-term Survey will start at 36 months following randomization. Skipping the data collection at 36 months would compromise the National Evaluation impact evaluation in several ways. Most seriously, it would make it nearly impossible to collect good data on training participation and the receipt of support services during training—particularly for the control group members. A preliminary review of the first round HPOG Impact study survey data collected from participants roughly 36 months after randomization showed that about 39 percent of the participants had additional training between the 15-month and 36-month follow-up survey. Without an Intermediate Follow-up Survey, that information would have been lost to researchers. These data are also essential for calculation of costs in support of the cost-benefit analysis. Second, skipping the data collection would jeopardize the ability to conduct a short assessment of basic literacy and numeracy skills at 36 months. Third, skipping the data collection would eliminate the ability for policy makers to determine whether there are intermediate signs that the HPOG 2.0 grants are achieving their purpose.
Descriptive evaluation. The evaluation team planned two rounds of telephone interviews with HPOG grantees. The first round of interviews took place in year 2 of the grant. The second round took place in year 4 of the grant. The descriptive evaluation will include only one round of participant interviews and telephone interviews with HPOG partners for the systems study. Less frequent data collection would prohibit timely collection of data about program implementation, costs, and systems over time.
Cost-benefit analysis study. The evaluation team plans only one round of cost data collection from HPOG grantees, completed between September 2019 and March 2020. This data collection is critical to estimating the cost of program inputs for the cost-benefit analysis study.
A7: Special Circumstances
There are no special circumstances for the proposed data collection.
A8: Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 27, 2018, Volume 83, Number 247, page 66715-66717, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment Y. During the notice and comment period, the government received no requests for information or substantive comments.
Consultation with Experts Outside of the Study
The HPOG 2.0 National Evaluation team had limited consultation with external experts in developing the instruments for this submission. The design of the National Evaluation Intermediate Follow-up Survey is based on the previously approved Short-term Follow-up Survey (OMB Control No. 0970-0462 approved in June 2018), along with other studies in ACF’s career pathways research portfolio, specifically the PACE 15-, 36-, and 72-month follow-up surveys (OMB #: 0970-0397); and the impact study of the first round of HPOG 15-, 36-, and 72-month surveys (OMB #: 0970-0394) to ensure consistency in the data collected across ACF’s entire career pathways research portfolio.
The external consultation focused primarily on how to measure basic skills, which is the focus of the phone-based Skills Assessment Pilot (Instrument 19). The experts consulted are listed in Exhibit A-6 below. This consultation took place in 2017.
Exhibit A-6: Experts Consulted Outside of the Study
Name |
Title/ Organization |
Contact Information |
HPOG 2.0 National Evaluation |
||
Meredith Larson |
Research
Analyst U.S. Department of Education |
[email protected] |
Stephen Provasnik |
Team Lead for International Activities Program National Center for Education Research Institute of Education Sciences U.S. Department of Education |
[email protected] |
Irwin Kirsch |
Director of the Center for Global Assessment Education Testing Services |
1-609-921-9000
|
The previously approved PAGES data items and the Tribal Evaluation materials were also developed in conjunction with substantive experts. The ICRs previously approved under OMB Control No. 0970-0462 in 2015 and 2017, respectively, provide more information on external consultation for those items. The instruments for the descriptive evaluation and cost-benefit analysis study did not require consultation with experts. The evaluation team includes staff nationally recognized for their expertise in cost-benefit analysis (Bob Lerman and Pam Loprest from The Urban Institute, Mary Farrell from MEF Associates). These team members also consulted with some of the non-tribal grantees to solicit their feedback on the content and burden associated with the cost survey.
A9: Incentives for Respondents
The evaluators plan to offer incentives to respondents for the National Evaluation impact and descriptive evaluations. Specifically, incentives are planned for the Intermediate Follow-up Survey, the phone-based Skills Assessment Pilot, and the participant in-depth interviews. The justification and incentive plans for each data collection activity are provided below.
Incentives—National Evaluation
Monetary incentives show study participants that the researchers appreciate their continued involvement in the HPOG 2.0 National Evaluation information collection activities. The HPOG 2.0 National Evaluation impact evaluation is a longitudinal panel RCT study, intended to follow selected impact evaluation participants for at least three years. Although there is little published evidence of the effectiveness of incentives in reducing nonresponse bias, it is well established that incentives strongly reduce study attrition (i.e., increase response rates) in panel studies such as the HPOG 2.0 National Evaluation.5,6 In accordance with OMB guidelines; the team took several factors into consideration when determining whether or not to use incentives.7 OMB published guidance to help agencies ensure that they maximized the quality of information collected.8,9 OMB defined “quality” in that guidance as “…the encompassing term, of which ‘utility,’ ’objectivity,’ and ‘integrity’ are the constituents.” Utility refers to how useful the information is to the intended audiences; objectivity focuses on both the presentation of the information collected and the substance of the information collection methods; and integrity takes into account the information collection protocols particularly as they relate to data security. This was taken into consideration when determining whether or not the use of incentives was necessary for the information collections that are the subject of this third revision of OMB Control Number 0970-0462.
The HPOG 2.0 National and Tribal Evaluation will ultimately provide ACF with information necessary to help objectively assess the HPOG Program and inform key stakeholders about lessons learned. It will provide policy makers with information to help make decisions about future reauthorization of the Program. The evaluation team and ACF worked closely to ensure that the previously approved procedures related to the data collected under this OMB Control Number met the OMB requirements for quality and transparency. Quality—from the perspective of integrity—is addressed in Section A10; while plans to ensure quality—with regard to objectivity and utility—are covered throughout Supporting Statement A and Supporting Statement B.
With regard to the use of incentives, the team focused on ensuring quality—particularly quality with regard to objectivity. The team took into account the key data quality risks posed by the longitudinal study design, our other efforts to reduce non-response bias, the burden on respondents, the complexity of the study design and panel retention over a 36-month period, and prior use of incentives for this study population previously approved under this OMB Control Number.
In a panel study such as the HPOG 2.0 National Evaluation, panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving sufficient sample size to maintain statistical power to detect meaningful effects in the analysis. Although low response rates do not necessarily lead to nonresponse bias, and it is at least theoretically possible to increase nonresponse bias by employing some techniques to boost response rates (Groves, 2006), most statisticians and econometricians involved in the design and analysis of randomized field trials of social programs agree that it is generally desirable to obtain a response rate close to 80 percent in all arms of the trial (Deke and Chiang, 2016). Based on the research team’s experience with differential response rates in the PACE and HPOG 1.0 data collection efforts—studies of similar scope and populations, the team believes that there is some risk that the HPOG 2.0 study will be sensitive to differential response rates, particularly if the team does not minimize attrition between the Short-term (15-month) and Intermediate (36-month) Follow-up Surveys. PACE had a differential response rate of 5.1 percentage points at 15-months and 5.0 percentage points at 36-months. In the HPOG 1.0 three-armed experiment, the response rate differential for the standard treatment vs. the control group was 7.2 percentage points and 6.6 percentage points for the enhanced treatment vs. the control group at 15 months. At the end of the 36-month follow-up data collection, the response rate differential was 3.4 percentage points between the standard treatment and control group, and 5.9 percentage points between the enhanced treatment and control group.
The treatment and control differential improved between the 15- and 36-month follow-up surveys for both PACE and HPOG 1.0 Impact studies. The team theorizes that incentives were a particularly powerful tool for maintaining a high response rate in the control group given that these sample members do not receive any (other) program benefits or services. Given the similarities between these studies, we expect similar response patterns for HPOG 2.0.
In most panel studies, response rates decline over follow-up rounds, potentially weakening the quality of the information collection—with regard to objectivity—as lower response rates could lead to increased nonresponse bias. The team has tried to minimize this expected decline and ensure a high response rate with a low treatment-control differential through the use of the previously approved welcome packet and repeated use of the previously approved participant contact update forms, combined with the provision of incentives, as discussed below. Through these tools the team hopes to address three goals:
Overcome participant mobility—over a long follow-up period, many study participants relocate multiple times, making it difficult to find them to complete a follow-up interview;
Reduce survey data collection costs— the more quickly interviewers can locate the respondent and complete an interview, the lower the costs per completed survey; and
Maintain participant engagement in a complex panel study—the ability to keep participants engaged in the research study for at least three years after enrollment is crucial to understanding long-term outcomes and the effectiveness of the HPOG Program.
The National Evaluation also provided incentives to those who participated in the in-depth interviews for the descriptive evaluation. The in-depth interviews were long and they often required participants to make arrangements for child care or transportation. The use of incentives to help offset the costs of participation is a well-established practice in social science research and program evaluation for both small-scale studies and sample surveys.
Previously Approved Incentives—National Evaluation
OMB previously approved the use of incentives in support of this study—the HPOG 2.0 National and Tribal Evaluations—under two prior revisions to OMB Control No. 0970-0462 (approved in June 2017 and June 2018). We drew upon experience with the incentives approved for the prior data collection components to specify the appropriate incentive amount for the additional HPOG 2.0 National and Tribal Evaluation data collection components proposed under the third modification request.
Exhibit A-7 provides a summary of the incentives approved under the HPOG 2.0 National and Tribal Evaluation to date.
Exhibit A-7: HPOG 2.0 National and Tribal Evaluation Previously Approved Incentives
Data Collection Activity |
Data Collection Instrument(s) |
Incentive Amount |
HPOG 2.0 National Evaluation |
||
Impact evaluation |
Welcome Packet (Instrument 5a, approved in June 2017) |
Participants received a non-monetary incentive (a portable cell phone charger), branded with the HPOG 2.0 study logo and toll-free study hotline phone number. |
Contact Update Request (Instrument 5b, approved in June 2017) |
The study team provides an incentive—a $5 gift certificate—for each contact update received from participants. |
|
Short-term (15-month) Follow-up Survey (Instrument 12, approved in June 2018) |
Respondents received a gift card valued at $40 upon completion of the survey. |
|
HPOG 2.0 Tribal Evaluation |
||
Tribal Evaluation |
Focus Group Guides
HPOG Program Completer Interviews
HPOG 2.0 Program Non-completer Interviews (Instruments 9, 10, and 11, approved in June 2017) |
Respondents receive a non-cash incentive valued at $50. |
Incentives Approved in July 2019 under the Third Revised Request for Clearance—National Evaluation
Impact evaluation. In the third revised request for clearance, approved in July 2019, the National Evaluation team requested and received permission to provide incentives for completion of the Intermediate Follow-up Survey and the phone-based Skills Assessment Pilot.
Three factors informed the study’s choice of the incentive amounts for survey respondents:
Respondent burden, both at the time of the interview and over the life of the study;
Costs associated with participating in the interview at that time; and
Other studies of comparable populations and burden—to help with the estimation of the appropriate incentive amount.
Given a target response rate of 80 percent for the
Intermediate Follow-up Survey, a minimum three-year follow-up period,
the incentive amounts previously approved for the HPOG 2.0 National
Evaluation impact evaluation Short-term Follow-up Survey (OMB control
number 0970-0462), and a similar study design and population in
OPRE’s Career Pathways studies (PACE and the first round HPOG
Impact studies; OMB control numbers 0970-0397 and 0970-0394
respectively), the evaluation team believes that incentives are
necessary to maximize the response rate to the impact evaluation’s
Intermediate (36-month) Follow-up Survey, balance the treatment and
control group response rate differential, and minimize attrition.
The impact evaluation team believes that an incentive of
$45 is the appropriate amount to help ensure that the evaluation
ultimately meets the quality targets defined by OMB. The Intermediate
Follow-up Survey sample is smaller than that for the Short-term
Follow-up Survey, so minimizing attrition is critical. Therefore, the
proposed incentive amount represents a modest $5 increase— from
$40 at the Short-term Follow-up Survey to $45 at the Intermediate
Follow-up Survey. These incentives are provided to help offset any
potential expenses incurred by the participant such as cell phone
minutes for those completed by telephone or childcare/transportation
costs for those completed in-person. The Intermediate Follow-up
Survey respondents will receive a gift card valued at $45.
Respondents will receive an email with instructions to log in to a
secure study portal where they can redeem a $45 gift card from their
choice of approved vendors. 10
Respondents who complete the interview in person and do not have an
email address will receive a card with redemption instructions.
A similar approach was used for the phone-based Skills Assessment Pilot. The evaluation team provided incentives to respondents who participated in this pilot data collection activity. Given that this was an exploratory effort, the length of the data collection activity was somewhat shorter and less burdensome to respondents than the follow-up surveys, and the participants who completed the assessment volunteered to do so, the evaluation team believed that an incentive valued at $25 was sufficient. Participants who completed the assessment received a gift card valued at $25 (to be redeemed in the same manner as described above). The evaluation team ensured that the participants who volunteered to participate in the assessment pilot were not part of the follow-up survey samples.
Following completion of the Short-term Follow-up Survey, a participant will continue to receive quarterly contact update requests in preparation for the Intermediate Follow-up Survey. Participants will receive the requests to update their contact information using the previously approved contact update form (Instrument 5b, approved in June 2017). Those who respond to the quarterly contact update requests will receive the $5 electronic gift certificate after each completed request (see OMB Supporting Statement A for OMB Control # 0970-0462, previously approved in June 2017 for details).
Descriptive evaluation. For the descriptive evaluation, the research team requested and received approval under the third revised submission to provide non-cash incentives for in-person participant interviews, in the form of a gift card. These interviews are estimated to take between 60 and 90 minutes to complete. Interview data are not intended to be representative in a statistical sense, in that they will not be used to make statements about the prevalence of experiences in the overall HPOG population. However, it is important to recruit participants with a range of background characteristics, to capture a variety of possible experiences with HPOG services. As all participants will be current or former HPOG program participants, the target population is, by definition, low income.
Although participation is voluntary and participants will have a choice of travelling to an interview or having an interviewer travel to them, there are potential burdens placed on study participants. These interviews may impose burden on the participant’s daily life. For instance, an interview that takes 60-90 minutes to complete could interfere in family commitments, result in substantial travel or child care costs, or result in unforeseen other expenses associated with research participation. Without offsetting the direct costs incurred by respondents for attending the interviews, such as arranging child care, transportation, or time off from paid work, the research team increases the risk that only those individuals able to overcome the financial barriers to attend will participate in the study, thus limiting the experiences the study is able to capture. For example, this may result in fewer participants with children participating in the study and thus the study would not reveal the experiences of attending the HPOG program while raising children.
Participants received a $40 gift card to account for expenses such as transportation and/or child care that may otherwise prevent their participation in the study. Studies have shown incentives’ effectiveness in increasing study participation among underrepresented populations such as individuals from low-income or low-education households, demographics of concern here. The amount of incentive is important in increasing participation rates. While research suggests little difference in the incentive amount when a participant is willing to participate, for participants who are less willing to participate, a larger incentive amount increases the likelihood of the potential participant agreeing to conduct the interview.11 Within the HPOG 2.0 National and Tribal Evaluation, OMB previously approved similar use of incentives for the HPOG 2.0 Tribal Evaluation (participant focus groups and interviews) in June 2017 under this OMB Control Number (0970-0462) in the amount of $50.The ACF PACE study (OMB control number 0970-0397, approval 08/19/2013), conducted with a similar population to that targeted by the current study, provided $40 incentive payments for interviews that lasted 60-90 minutes and 84% of those asked, participated. We believe $40 is a reasonable amount for the time and cost associated with participation in these data collection activities, but is not so high as to appear coercive for potential participants.
To receive the $40 gift card, participants received an email with instructions to log in to a secure study portal where they can redeem the gift certificate to one of the approved vendors (see procedures for redeeming procedures described under the Intermediate Follow-up Survey).
A10: Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
For all the non-tribal grantees participating in the national evaluation, study participants must provide an SSN in order to enroll in the program. The previously approved consent forms (Attachment B: National Evaluation Informed Consent Form A (Lottery Required) and National Evaluation Informed Consent Form B (Lottery Not Required); and Attachment B2: Tribal Informed Consent Form A (SSNs)) clearly state how SSNs will be used in the evaluation. Those same statements are reflected in the new verbal consent form versions—Attachment B: National Evaluation Informed Consent Form C (Lottery Required)_Verbal and National Evaluation Informed Consent Form D (Lottery Not Required)_Verbal; and Attachment B2: Tribal Evaluation Informed Consent Form C (SSNs)_Verbal.
As specified in the evaluator’s contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. All project and grantee staff members with access to PAGES sign a New User Data Security Agreement and they undergo training on data privacy and security. Grantees participating in the National Evaluation that do not have their own Institutional Review Board (IRB) or Federalwide Assurance Number (FWA) sign individual investigator agreements, which allows the protection under Abt’s FWA. Grantees participating in the Tribal Evaluation that do not have their own IRB or FWA sign individual investigator agreements, which will allow them protections under NORC’s FWA.
As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor has submitted a plan for minimizing to the extent possible the inclusion of sensitive information on paper records (e.g., the consent forms) and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
None of the respondents that participate in interviews will be identified in any report or publication of this study or its results; their participation will be voluntary; and their information will be kept private. This information will be provided verbally to interview respondents in both studies, and verbal consent will be requested.
As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to HPOG 2.0 participants by grantees:
Research is being conducted to see if and how HPOG 2.0 makes a difference in people’s lives by helping them complete training and get healthcare jobs. This program and research are funded by HHS, and HHS may fund other research on this program in the future.
In this program, grantees will collect some personal information from individuals, such as their name, date of birth, Social Security number, and involvement in other programs.12 The researchers studying the program for the government also need this information. Researchers will use data security procedures to keep all of the study data private and to protect individuals’ personal information. All of the information collected for the program or for the research studies will be kept completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
Researchers may contact applicants at grantees participating in the impact evaluation in the future. Individuals may refuse to answer any of their specific questions at any time.
A.10.2 PAGES
The PAGES system has both a System of Records Notice (SORN)—09-80-0361 OPRE Research and Eval Project Recs—and Authorization to Operate (ATO). For further description of security measures related to PAGES, please see the original submission approved under OMB Control No. 0970-0462 in August 2015.
To ensure data security and enhance data quality, the Intermediate Follow-up Survey data collection will be done using Computer Assisted Personal Interviewing or CAPI technology using the Confirmit CAPI System. The Confirmit CAPI System has the following security features:
Data on the CAPI console is encrypted with Rijndael algorithm (256 bit key).
CAPI data transfers use Web Services Enhancements (WSE 3.0) for security. The messages sent and received from the console are encrypted. WSE 3.0 provides AES128 + RSA 1.5 as default algorithms for symmetric encryption and key-wrap. The research team has also implemented Secure Conversation with an X509 certificate (which uses 1024 bit key).
In addition to the standard security features offered through the CAPI software, the research team has implemented the following enhancements:
Use of PGP whole disk encryption on all CAPI laptops and tablets, and
The file transfers are made to servers running SSL.
As surveys are completed, data will be transferred from the CAPI system to the study’s database. Transfer to the database will be done in a secure manner, using a FIPS-certified encryption algorithm. Once the Intermediate Follow-up Survey data collection is complete, all survey records will be transferred to the analytic database, stored on Abt Associates’ secure Analytical Computing Environment (ACE3), the FISMA moderate server, where most analyses will be conducted. ACE3 currently provides:
A secure, isolated environment utilizing Amazon's FedRAMP Moderate accredited services as infrastructure
Secure server and application configurations that meet NIST SP 800-53 Revision 4 FISMA Moderate standards where appropriate, with compliant policies and procedures
FedRAMP Moderate accredited file transfer services for moving data in and out of the system
Fully redundant architecture where possible, with architected scalability and elasticity to deal with the storage and processing of large data sets by increasing available memory, CPU, or disk space availability
Enhanced monitoring by AWS CloudWatch and the leading third party vendor for Log Monitoring: Dell SecureWorks
Enhanced availability and backups using native AWS services
The analytic databases are designed to limit access only to authorized users with levels of access commensurate with each person’s role on the project. PII will be separated from the rest of the information and stored in a separate folder which only the project director (PD), the deputy project director (DPD), and a small number of designated analysts (5-10 people) will be able to access. The de-identified survey data will be stored in folders that will be accessible by the PD, DPD, PI, the director of analysis, and a small team of other statisticians, economists, and analysts working on the evaluation. Only tabular data and other high-level summaries (such as regression coefficients) will be stored in the general servers of the Prime contractor (Abt Associates), shared with subcontractors via email, and eventually published. The web server hosting the database is maintained in a secure facility with power back up, network redundancy, and system monitoring. In addition, daily back up of the server is maintained at the data center and an off-site location. The database and website are password protected, and access is provided only after user authentication. After the reports are published, the data will be archived. The specific plans for archiving the data and decisions on whether or not to create public or restricted access datasets are to be determined. The archiving procedures will ensure that de-identified data are stored securely.
For participant-level data collected from both survey data and corresponding administrative data from the National Student Clearinghouse (NSC), computer security will be maintained by individual passwords and folder permissions which limit access to files to only those project staff members who require access to these files and have appropriate permission to do so.
All administrative data from the National Directory of New Hire (NDNH) will reside on ACF secure servers. Only Abt staff members granted ACF security clearance will have access to the data on ACF loaned laptops and the secure folder. All analysis of NDNH data will be conducted on ACF’s secure server. All survey information and analytic data will be accessed by the respondent’s study ID number. No personally identifiable information will be maintained on paper.
A11: Sensitive Questions
This section summarizes the sensitive questions asked of respondents under the HPOG 2.0 National Evaluation for the instruments in this submission and provides guidance on where to find sensitive questions on the other previously approved instruments. It then provides an overview of the sensitive questions contained in the National Evaluation impact evaluation’s Intermediate Follow-up Survey and the National Evaluation descriptive evaluation’s in-depth participant interviews.
PAGES Participant-Level Baseline Questions
For a description of sensitive questions related to PAGES, please see the original submission approved under OMB Control No. 0970-0462.
HPOG 2.0 Tribal Evaluation
For a description of sensitive questions in previously approved Tribal Evaluation instruments, please see the first revised submission approved under OMB Control No. 0970-0462, approved in June 2017.
HPOG 2.0 National Evaluation
Impact evaluation. The questions included in the Short-term Follow-up Survey that could be considered sensitive are described in the previously approved request for clearance under OMB Control Number 0970-0462, approved in June 2018.
The Intermediate Follow-up Survey includes several questions about overall physical health, income, receipt of government benefits, fertility, and household composition, items that some respondents may consider sensitive. As it is hoped that HPOG 2.0 will have favorable impacts in all these areas, failure to ask any of them would limit the findings of the evaluation. Interviewers will remind study members during the interview that they may refuse to answer individual items. Interviewers will also remind study members that their responses will be kept private to encourage their candid responses.
The phone-based Skills Assessment Pilot does not include any sensitive questions.
Descriptive evaluation. Several questions in the HPOG 2.0 National Evaluation participant interviews may be considered sensitive by some program participants. These questions ask about participant and family finances, needs, and types of supportive services received, including academic, social, and employment related. These questions are necessary because supportive services are a key component of the HPOG Program. Data collected will be used to identify how HPOG programs assess student needs and what types of services they are offering as part of their program. Interviewers will inform participants that their participation is voluntary, that they may decline to answer any question that they wish, that their information will be kept private, and they will not be identified in any report or publication of this study or its results. The second round telephone interviews and systems study interviews with HPOG staff, management, and partners do not include any sensitive questions.
Cost-benefit analysis study. The program cost survey will collect only organization-level information and does not include any sensitive questions.
A12: Estimation of Information Collection Burden
The burden estimates described in this section were approved in July 2019. The non-substantive changes requested in this submission do not require any changes to those estimates.
This section provides information on the information collection burden estimates in five categories:
The total previously approved burden;
The total burden remaining from the previously approved information collections;
The additional burden estimates for previously approved information collections;
The burden for the newly requested information collection; and
A summary of all burden remaining.
A12.1 Previously Approved Information Collections
Total Burden Previously Approved
The previously approved burden estimates included: (1) burden on grantee staff members who enter grantee-level and ongoing participant-level data into PAGES to complete the HPOG PPRs; (2) burden on HPOG applicants to complete the baseline questions; and (3) burden on grantee staff who enter the baseline data into PAGES. It also includes burden for various data collection activities under the National and Tribal Evaluations.
The total burden for all previously approved instruments was estimated to be 18,614 hours annually, or 55,842 hours total over three years.
Burden Remaining from Previously Approved Information Collection
Estimated burden remaining to continue use of the previously approved instruments is 7,783 hours annually or 23,349 total hours over the next three years. Exhibit A-8 shows the remaining hourly and cost burden by instrument. The data collection using Instruments 2, 3 and 4 (First-Round Telephone Interview and Screening Guides, and In-person Implementation Interviews) for the National Evaluation descriptive study and Instrument 5a (Welcome Packet) under the impact evaluation is complete so there is no remaining burden reflected for them.
Exhibit A-8: Burden Remaining from Previously Approved Information Collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
||||
Instrument 1: PAGES Grantee- and Participant-Level Data Collection (all grantees) |
96 |
32 |
2 |
31.75 |
2,032 |
$28.29 |
$57,485.28 |
||||
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at non-tribal grantees) |
14,800 |
4,934 |
1 |
.5 |
2,467 |
$3.94 |
$9,719.98 |
||||
Instrument 1: PAGES Participant-Level Baseline Data Collection (participants at Tribal grantees) |
1,288 |
430 |
1 |
.25 |
108 |
$3.94 |
$425.52 |
||||
Instrument 5b: HPOG 2.0 National Evaluation Letter and Participant Contact Update Form |
9,000 |
3,000 |
3 |
.1 |
900 |
$10.15 |
$9,135 |
||||
Instrument 12: HPOG 2.0 National Evaluation Short-term Follow-up Survey |
5,850 |
1,950 |
1 |
1 |
1,950 |
$10.15 |
$19,792.5 |
||||
HPOG 2.0 Tribal Evaluation |
|||||||||||
Instrument 6: HPOG 2.0 Tribal Evaluation Grantee and Partner Administrative Staff Interviews |
70 |
24 |
1 |
1 |
24 |
$28.29 |
$678.96 |
||||
Instrument 7: HPOG 2.0 Tribal Evaluation Program Implementation Staff Interviews |
100 |
34 |
1 |
1.5 |
51 |
$28.29 |
$1,442.79 |
||||
Instrument 8: HPOG 2.0 Tribal Evaluation Employer Interviews |
60 |
20 |
1 |
.75 |
15 |
$50.99 |
$764.85 |
||||
Instrument 9: HPOG 2.0 Tribal Evaluation Program Participant Focus Groups |
270 |
90 |
1 |
1.5 |
135 |
$10.15 |
$1,370.25 |
||||
Instrument 10: HPOG 2.0 Tribal Evaluation Program Participant Completer Interviews |
200 |
67 |
1 |
1 |
67 |
$10.15 |
$680.05 |
||||
Instrument 11: HPOG 2.0 Tribal Evaluation Program Participant Non-completer Interviews |
100 |
34 |
1 |
1 |
34 |
$10.15 |
$345.10 |
||||
Estimated Annual Burden Previously Approved |
7,783 |
|
$ 101,840.28
|
||||||||
A12.2 Additional Burden for Previously Approved Information Collection
The final sample size for the Short-term Follow-up Survey is 13,118. A response rate of 80 percent would yield 10,494 completed interviews. This is 94 additional completed interviews than what was previously approved in June 2018. The additional burden is 31 hours annually over the next three years, with a cost of $314.65 per year.
ACF seeks to increase the burden for the previously approved contact update form (Instrument 5b) to continue collecting updated contact information for the intermediate follow-up survey impact evaluation study participants. The increased burden will allow the evaluation team to continue to send contact update requests to those participants beyond the 36-month follow up period. Extending the contact update data collection beyond the 36 month follow-up period would ensure that a long term follow-up survey effort would be feasible, should ACF decided decide to fund one. (See Section A15 for more detail on change in burden). Extending the contact update data collection increases the previously approved burden by 275 hours annually over three years, and adds $2,792.27 to the total cost.
Exhibit A-9 shows the additional burden estimates in both hours and cost, associated with the increased survey sample size and extended contact update efforts.
Exhibit A-9: Additional Burden for Previously Approved Information Collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Instrument 12: HPOG 2.0 National Evaluation Short-term Follow-up Survey |
94 |
31 |
1 |
1 |
31 |
$10.15 |
$314.65 |
Instrument 5b: HPOG 2.0 National Evaluation letter and participant contact update form |
2,750 |
917 |
3 |
.1 |
275 |
$10.15 |
$2,791.25 |
Estimated Additional Annual Burden Previously Approved |
306 |
|
$3,105.90 |
||||
A12.3 Newly Requested Information Collections
Exhibit A-10 presents the reporting burden in both hours and cost for National Evaluation impact evaluation participants and those who complete the data collection items for the descriptive evaluation and cost-benefit analysis study that are included under this information collection request.
Exhibit A-10: Burden for Newly Requested Information Collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
HPOG 2.0 National Evaluation |
|||||||
Instrument 13: Screening tool for identifying respondents for second-round telephone interviews |
38 |
13 |
1 |
.5 |
7 |
$28.29 |
$198.03 |
Instrument 14: Second round telephone interview protocol for non-tribal HPOG grantee staff and partners |
190 |
63 |
1 |
1.25 |
79 |
$28.29 |
$2,234.91 |
Instrument 15: Program operator interview guide for the systems study |
16 |
5 |
1 |
1.25 |
6 |
$28.29 |
$169.74 |
Instrument 16: Partner interview guide for the systems study |
112 |
37 |
1 |
1 |
37 |
$28.29 |
$1,046.73 |
Instrument 17: Participant interview guide |
140 |
47 |
1 |
1.33 |
63 |
$10.15 |
$639.45 |
Instrument 18: Intermediate Follow-up Survey for the National Evaluation impact study |
4,000 |
1,333 |
1 |
1 |
1,333 |
$10.15 |
$13,529.95 |
Instrument 19: Phone-based Skills Assessment Pilot Instrument |
300 |
100 |
1 |
0.75 |
75 |
$10.15 |
$761.25 |
Instrument 20: Program cost survey |
38 |
13 |
1 |
7 |
91 |
$28.29 |
$2,574.39 |
Estimated Annual Burden Total |
1,691 |
|
$21,154.45 |
||||
A12.4 Total Burden under OMB # 0970-0462
Exhibit A-11 shows the estimated annual respondent burden over the next three years is 9,780 hours. This represents the total burden remaining from the previously approved information collection, the additional burden hours for previously approved information collection due to a higher number of projected completed Short-term Follow-up Surveys, and the burden for the new information collection.
Exhibit A-11: Total Burden Remaining Under OMB #0970-0462
Instrument |
Annual Burden Hours |
Burden Remaining from Previously Approved Information Collection |
7,783 |
Additional Burden for Previously Approved Information Collection |
306 |
Burden for New Information Collection |
1,691 |
Total Annual Burden Hours |
9,780 |
Total Annual Cost
To compute the total estimated annual cost reported in Exhibits A-8, A-9 and A-10 evaluators used the average wage for HPOG 1.0 participants employed at program intake ($10.64) and multiplied that by the proportion of those working at intake (0.37) for an average hourly total of $3.94. Evaluators believe the HPOG 1.0 data provide an accurate basis for estimating wages for HPOG 2.0 study participants for the previously approved information collection under PAGES. The baseline wage was appropriate for the original HPOG Next Generation submission as the PAGES system collects wage information at the time of enrollment. Since, this is a job training program the cost in the burden table in Supporting Statement A was revised to reflect the loaded federal minimum wage. The loaded federal minimum wage was used in the previously approved information collection requests for the HPOG 1.0 15- and 36-Month Follow-up Surveys and the PACE 15 and 36-Month Follow-up Surveys (OMB Nos. 0970-0394 and 0970-0397 respectively). For the cost to grantees and partner organizations data collection efforts, the total burden costs were multiplied by the average hourly wage, according to the Bureau of Labor Statistics, National Compensation Survey, 2010 ($28.29/hour). 13 The average hourly wage for the employer interviews is based on Bureau of Labor Statistics code 11-9111, Medical and Health Services Managers ($50.99). The total annual cost burden of collecting this new information is $21,154.45. The evaluation team estimates that the annual costs for the remaining previously approved data collection is $101,840.28 over the next three years. The cost associated with the additional previously approved information collection is $3,105.90. The total annual cost burden for all efforts combined is estimated at $126,091.46.
A13: Cost Burden to Respondents or Record Keepers
Not applicable. The proposed information collection activities do not place any new capital cost or cost of maintaining capital requirements on respondents.
A14: Estimate of Cost to the Federal Government
The evaluation team estimated the costs to complete the National and Tribal Evaluation based on their (1) experience with the first round HPOG 1.0 evaluations (OMB number 0970-0394); (2) experience with the Pathways for Advancing Careers and Education (PACE) study (OMB number 0970-0397), particularly for the Participant Interviews; and (3) experiences conducting longitudinal evaluations of hard-to-locate, low-income populations. The team carefully examined the different data collection approaches available to them and determined which options were most effective in their experiences on other similar studies. The cost estimates developed for the HPOG 2.0 National and Tribal Evaluations reflect our best estimates on the costs to conduct data collection with this population, and the most efficient methods for doing so.
The total cost for the data collection activities under the previously approved third revised submission are $5,477,757. Of that total, $3,630,491 is for the National Evaluation impact evaluation’s Intermediate Follow-up Survey and the phone-based Skills Assessment Pilot. The costs for the National Evaluation descriptive evaluation second-round telephone interviews and participant interviews were $961,184. The costs for the systems study interviews were $199,173 and the cost-benefit analysis study costs were $686,910.
The costs for all of the prior previously approved information collection requests under OMB Control Number 0970-0462 were $13,925,591. These costs included $8,473,750 for the National Evaluation impact evaluation’s Short-term Follow-up Survey, $1,788,164 for the National Evaluation descriptive evaluation, $418,236 for the increased burden for the previously approved instruments (see A12.2 and A15 for more detail), and $1,225,193 for the Tribal Evaluation, for a total of $11,905,343, plus $2,020,248 for the original submission.
Thus, the total costs to the Federal government for all information collections under this OMB control number (0970-04562) are $19,403,348.
Exhibit A-10 below summarizes the costs to the Federal Government specifically associated with this information collection over the next three years. The table shows the three-year costs for four activity categories:
Ongoing PAGES data collection and reporting;
Design tasks (including instrument development, pretesting, OMB, and IRB activities);
Field Work (including impact study follow-up surveys, descriptive study interviews, participant interviews, focus groups, phone-based skills assessment pilot, and program cost surveys plus programming and training activities related to each); and
Analysis and Initial Reporting.
The table provides details on the number of total labor hours, the number of full-time equivalent staff persons they equate to, and loaded labor costs, plus operational expenses and other costs for each activity category.14 Operational expenses and other costs include such items as printing, equipment, overhead, shipping, travel, incentives, etc.
Exhibit A-11: Total Three Year Costs to the Federal Government under OMB #0970-0462 by Activity
Activity |
Labor Hours and Full-time Equivalents (FTEs) |
Labor Costs |
Operational Expenses and Other Costs |
Total Costs |
Ongoing PAGES data collection and reporting |
|
$191,922 |
$1,501,703
|
$1,693,625 |
Design |
|
$583,657 |
$380,774
|
$964,431 |
Field Work |
|
$5,174,490 |
$4,439,031
|
$9,613,521 |
Analysis and Initial Reporting |
|
$1,178,398 |
$703,795
|
$1,882,193 |
Total Costs Over Three Years |
|
$7,128,467 |
$7,025,303
|
$14,153,770 |
Annual Cost |
|
$2,376,155 |
$2,341,768
|
$4,717,923 |
A15: Change in Burden
This is a request for non-substantive changes for new alternative versions of the tribal and non-tribal evaluation consent forms (Attachment B, National Evaluation Informed Consent Form C (Lottery Required)_Verbal, National Evaluation Informed Consent Form D (Lottery Not Required)_Verbal; Attachment B2: Tribal Evaluation Informed Consent Form C (SSNs)_Verbal; and Attachment B3: Tribal Evaluation Informed Consent Form D (Unique Identifiers)_Verbal). These forms mirror the consent forms previously approved under OMB #0970-0462 (Attachment B: National Evaluation Informed Consent Form A (Lottery Required), National Evaluation Informed Consent Form B (Lottery Not Required); Attachment B2: Tribal Evaluation Informed Consent Form A (SSNs); and Attachment B3: Tribal Evaluation Informed Consent Form B (Unique Identifiers))), but they allow for verbal consent rather than written consent. These changes have no impact on respondent burden.
The July 2019 approval included an additional 94 burden hours associated with the previously approved Short-term Follow-up Survey. The original estimated number of completes was 10,400, but the revised expectation is for 10,494 completed interviews. The increase is due to the fact that the final enrollment targets were still estimates at the time that the package for approval of the Short-term Follow-up Survey (Instrument #12) was submitted for review. The total enrollment for the cohorts included in that sample was slightly higher than 13,000 estimated, thus there will be slightly more completed interviews.
Under the first round HPOG 1.0 impact study ACF added a long-term follow-up study to measure impacts six years after random assignment (approved under OMB Control # 0970-0394). ACF may opt to do the same for the second round of HPOG grants as part of the HPOG 2.0 National Evaluation impact study. In order to ensure that the interviewers can reach the study participants for any subsequent information collection efforts, it is important to maintain updated contact information once the initial 36-month follow-up period ends. Therefore, ACF seeks approval increase the burden for the previously approved contact update form (Instrument 5b). The contact update form collects updated participant contact information for impact evaluation participants (treatment and control) during the follow-up period. This form is sent every three months. The increased burden would only be for those study participants selected to participate in the Intermediate Follow-up Survey. Should ACF decide to do a longer term follow-up survey, that instrument would be submitted for approval under a future request for clearance.
A16: Plan and Time Schedule for Information Collection, Tabulation and Publication
16.1 Analysis Plan
Exhibit A-12 summarizes the primary domains covered in the Short-Term and Intermediate Follow-up Survey instruments and provides a brief discussion of how they will be used.
Exhibit A-12: Domains for HPOG 2.0 Short-Term and Intermediate Follow-up Survey Instruments
Domain |
Notes |
Uses |
A. Training and employment history |
Dates of every school and job spell since randomization. Reasons for no school/job during ever (Short-term Follow-up Survey only). Careful probing for simultaneous study and work as well as multiple job holding. Dates of every school and job spell since randomization. |
|
B. School Experiences |
For each school spell:
|
|
C. Credential attainment and education/career goals |
Mostly about credentials, both those issued by schools and those issued by other authorities. |
|
D. Terms of employment and conditions at current/last job |
Occupation, scheduling, hourly wage rate, typical hours, benefits, other quality measures. |
|
E. Household composition |
Living arrangements, counts of adults and children, family formation, child bearing. |
|
F. Income and financial well-being |
Includes personal and household participation in government anti-poverty programs as well as income; includes questions on financial well-being and material hardship. |
|
G. 21st Century Skills |
Use of computer, literacy and numeracy skills at work and in everyday life; self-directed learning
|
|
H. Parent report on child well-being |
Child’s education status, child supervision, family routines, educational goals and support, and child outcomes |
|
I. Respondent and Secondary Contact Information |
Address, phone, and email information for the respondent and up to three people who know how to reach the respondent |
|
J. Basic Skills Assessment |
Basic skills (literacy and numeracy) assessment battery (Intermediate Survey Only) |
|
The HPOG 2.0 National Evaluation team will produce several reports using the data collected for the descriptive evaluation. The reports will include:
Descriptive Evaluation Report. This report will summarize the information on program implementation features, challenges, and best practices using the descriptive evaluation interviews, site visit data, and data from PAGES. This report will include the implementation, outcome, and systems studies. The evaluation team will produce a series of briefs from the participant interview data. The evaluation will also use results from the implementation study to produce short case study reports on focus areas of specific interest to ACF.
Impact Evaluation Reports. Findings from the implementation study will inform the analysis in the evaluation’s impact evaluation reports. The evaluation is expected to produce reports on results based on the Short-term (15-month) and Intermediate (36-month) Follow-up Surveys and associated administrative data analysis. The team will also produce a longer-term (60-months post randomization) impact report using administrative data only.
Phone-based Skills Assessment Pilot. We will use factor analysis and item response theory (IRT) to identify a subset of the developed items that satisfy three goals: (1) Even variation in level of difficulty; (2) High reliability of a scale based on the smaller set of items; and (3) High correlations of the reduced scale with earned credentials and with the Programme for International Assessment of Adult Competencies (PIAAC) questionnaire items.
Since the pilot sample will not contain any members of the control group, it will not be possible to estimate the effect of HPOG 2.0 on these items during the pilot stage. However, once the subset of items to use in the Intermediate Follow-up Survey is identified they could be used to generate estimates of the impact of the program on academic/career skills. The evaluation team will first discard any items that are either always or never answered correctly, and then use factor analysis to separate the items into sets that seem to be measuring different latent skills (presumably vocabulary and arithmetic skills) and item response theory to rank the difficulty of the items within each set. The evaluation team anticipates selecting about 12 vocabulary items and 12 arithmetic items for inclusion in the Intermediate Follow-up Survey. Each group of 12 will contain four items of (relatively) low difficulty, four items of (relatively) medium difficulty, and four items of (relatively) high difficulty.
Finally, the evaluation team anticipates selecting the items that best satisfy these goals using nonlinear programming, aiming to maximize the correlation of the reduced set with the full set for the latent skill. This is subject to the constraint that four easy items, four moderately difficult items, and four difficult items are identified. After this, the team will correlate the reduced set with earned credentials and the PIAAC questionnaire items. If some excluded items have higher correlations with these validated scales than the tentative reduced scales, we will explore strategic substitutions.
The HPOG 2.0 Tribal Evaluation will use a systematic approach to analyze the data obtained through the interviews and focus groups conducted during and following annual site visits. The evaluation team will use NVivo software to store and analyze the large volume of data collected over the course of the evaluation. NVivo will be used to develop a coding scheme for analyzing these data. The coding scheme will be organized around evaluation topic areas derived from the evaluation questions. The coding scheme will be applied to all data and emergent key themes relating to evaluation topic areas will be identified.
The HPOG 2.0 Tribal Evaluation team will prepare a variety of reports, including site visit reports, practice briefs, and a final report.
Site Visit Reports. These reports will be developed after each annual site visit and summarize the findings from the interviews and focus groups.
Practice Briefs. Practice briefs will be shorter documents that highlight findings from the evaluation and share lessons learned.
Final Report. The final report will reflect the aggregated analysis of all qualitative and quantitative data collected throughout the evaluation
PAGES will automatically generate quantitative measures for the federally required semi-annual Program Performance Reports, which will include aggregated participant-level data, and will also store narrative-based grantee-level performance information. ACF will use these tables when preparing reports to Congress on the HPOG initiative. The PAGES team will produce a number of reports using data collected, including the six semi-annual PPRs and four annual reports.
The informed consent documents, whether signed or agreed to verbally, by the National and Tribal Evaluation study participants include permission for ACF and the evaluation team to archive the data. The specific plans for creating a public or restricted access dataset are still to be determined, but no data archiving will begin before the reports are published.
16.2 Time Schedule and Publications
The National Evaluation descriptive evaluation data collection began in July 2017, following OMB approval on the previous package. Contact updates for those participants in the National Evaluation impact evaluation sample began in November 2017 and will continue throughout the follow-up period. The National Evaluation impact study Short-term Follow-up Survey data collection began in October 2018 and will continue through the end of 2019. The Tribal Evaluation data collection began in October 2017. PAGES data collection will occur as individuals apply for the programs and enrollees receive training and services throughout the next three years of the grant period. Exhibit A-12 presents an overview of the project schedule for information collection.
Exhibit A-13: Project Schedule for Data Collection, Analysis, and Publication
Task |
Timing |
HPOG 2.0 National Evaluation: Descriptive Evaluation |
|
Descriptive evaluation data collection (includes, systems, and program implementation) |
June 2017-December 2019 |
National evaluation descriptive study site visits |
Fall 2018 |
Descriptive evaluation Analysis Plan |
Spring 2020 |
Descriptive evaluation Report (including implementation study, outcome study, and systems study) |
Final December 2020 |
HPOG 2.0 National Evaluation: Impact Evaluation |
|
Contact update mailing for survey sample |
Quarterly beginning 3 months after random assignment (November 2017-Summer 2022) |
Short-term (15-Month) Follow-up Survey Data Collection |
October 2018-November 2019 (15 months after randomization for participants enrolled between March 2017 and February 2018) |
Intermediate (36-Month) Follow-up Survey Data Collection |
Beginning in September 2020 (approximately 36 months after randomization for selected participants enrolled between March 2017 and February 2018) |
Draft Short-term Impact Report to ACF |
Summer 2020 |
Data Archiving |
After publication of the Short-term Impact Report |
Draft Intermediate Impact Report to ACF |
Spring 2022 |
Data Archiving |
After publication of the Intermediate Impact Report |
Draft Longer-term Impact Report to ACF |
September 2024 |
HPOG 2.0 National Evaluation: Cost-benefit analysis study |
|
Program cost data collection |
Fall 2019 – Spring 2020 |
Draft Cost-Benefit report to ACF |
Spring 2022 |
HPOG 2.0 Tribal Evaluation |
|
Site visits to tribal grantees (1/year) |
Annually spring/summer of 2017, 2018, 2019, 2020 |
Conduct data analysis |
2017-2021 |
Develop Practice Briefs |
Annually September 2017-September 2021 |
Develop Final Report |
September 2021 |
HPOG 2.0 National and Tribal Evaluation Participant Accomplishment and Grantee Evaluation System (PAGES) |
|
PAGES grantee-level and ongoing participant-level data collection |
September 2015 – September 2020 |
6 Semi-annual PPRs |
September 2015 – September 2020 |
Two annual reports |
September 2015 – September 2020 |
A17: Reasons Not to Display OMB Expiration Date
All instruments created for the HPOG 2.0 National Evaluation and Tribal Evaluation will display the OMB approval number and the expiration date for OMB approval.
A18: Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 Tribal participants are only part of an outcomes study. They have the ability to opt out of the study and still receive services.
2 The Office of Child Support Enforcement (OCSE), which maintains the NDNH data, typically requires written consent to allow the evaluation contractor to match study participant identifiers to the NDNH data. OCSE has agreed to amend the study’s Memorandum of Understanding to temporarily allow the collection of NDNH data with verbal consent.
3 References to “counseling” refer to services such as tutoring, academic advising, financial aid advising, career counseling, job search or placement assistance, and case management.
4 Additional data sources will be used to answer the research questions, beyond those listed here. Only the data sources relevant to this submission are listed.
5 The HPOG 2.0 impact evaluation is a panel study. The three primary points of data collection are the previously approved Baseline Intake Form administered immediately prior to randomization, the previously approved Short-term Follow-up Survey, initiated 15 months after randomization, and the Intermediate Follow-up Survey, projected to begin 36 months after randomization (for which clearance is requested in this package). The impact evaluation will also include a longer-term follow-up evaluation, using administrative data only.
6 See Chapter 12 of Lynn (2009), in particular, section 12.5 that reviews the effects of incentives in several prominent panel studies.
7 See page 69, questions 75 and 76, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf
8 See updated Information Quality Act guidelines (https://www.whitehouse.gov/wp-content/uploads/2019/04/M-19-15.pdf)
9 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, 67 FR 8452 (Feb. 22, 2002), available at: https://www.federalregister.gov/documents/2002/02/22/R2-59/guidelines-for-ensuring-and-maximizing-the-quality-objectivity-utility-and-integrity-of-information
10 In accordance with HPOG funding requirements, the incentives can be redeemed only through vendors that do not sell alcohol, tobacco, firearms or other entertainment.
11 Kelly, B., Margolis, M., McCormack, L., LeBaron, P.A., Chowdhury, D. “What Affects People’s Willingness to Participate in Qualitative Research? An Experimental Comparison of Five Incentives.” Field Research. Volume: 29 issue: 4, page(s): 333-350
12 Two Tribal grantees will not collect social security numbers from some or all of their participants. A unique identifier will be assigned for these participants. Two versions of the Tribal informed consent forms were developed, one that includes social security numbers and one for grantees using unique identifiers.
13 Source: Bureau of Labor Statistics, National Compensation Survey, 2010: Combined average hourly wage across education training and library occupations and community and social services occupations.
14 The number of full-time equivalents (FTEs) is calculated by dividing the total number of labor hours by 2,080 workable labor hours per year.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Katheleen Linton |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |
© 2024 OMB.report | Privacy Policy