Supporting Statement for OMB Clearance Request
Part A
National Implementation Evaluation of the
Health Profession Opportunity Grants (HPOG) to Serve TANF Recipients and Other Low-Income Individuals and HPOG Impact Study
0970-0394
April 24, 2013
Revised July 26, 2013
Submitted by:
Office of
Planning,
Research & Evaluation
Administration for Children & Families
U.S. Department of
Health
and Human Services
Federal Project Officers
Molly Irwin and Mary Mueggenborg
Table of Contents
A.1 Necessity for the Data Collection 2
A.1.2 Legal or Administrative Requirements that Necessitate the Collection 2
A.1.5 Universe of Data Collection Efforts 4
A.2 Purpose of the Survey and Data Collection Procedures 6
A.2.1 Overview of Purpose and Approach 6
A.2.2 Data Collection Process 7
A.2.3 Who Will Use the Information 8
A.2.4 Instrument Item-by-Item Justification 8
A.3 Improved Information Technology to Reduce Burden 14
A.4 Efforts to Identify Duplication 14
A.4.2 Coordination and Streamlining of Study Efforts 15
A.5 Involvement of Small Organizations 15
A.6 Consequences of Less Frequent Data Collection 16
A.8 Federal Register Notice and Efforts to Consult Outside the Agency 16
A.8.1 Federal Register Notice and Comments 16
A8.2 Consultation with Experts Outside of the Study 17
A.10 Privacy of Respondents 18
A.12 Estimation of Information Collection Burden 20
A.12.1 Baseline Data Collection Already Approved 20
A.12.2 Current Information Collection Request 20
A.12.3 Total Burden Hour Request 22
A.13 Cost Burden to Respondents or Record Keepers 22
A.14 Estimate of Cost to the Federal Government 22
A.16 Plan and Time Schedule for Information Collection, Tabulation and Publication 23
A.16.2 Time Schedule and Publications 24
A.17 Reasons not to Display OMB Expiration Date 24
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 25
Appendices
A. HPOG Logic Model
B. HPOG-NIE Sampling Questionnaire for the HPOG Surveys
C. HPOG-NIE Follow-Up Phone Call Protocol for the Stakeholder/Network Survey
D. HPOG-NIE Grantee Survey
E. HPOG-Impact Implementation Interview Guides
F. HPOG-NIE Management and Staff Survey
G. HPOG-NIE Stakeholder/Network Survey
H. HPOG-NIE Employer Survey
I. HPOG-Impact and HPOG-NIE 15-month Participant Follow-Up Survey
J. HPOG-Impact 15-month Control Group Member Follow-Up Survey
K. OMB 60-Day Notice
L. Consent Forms for Participant Surveys
M. Information on the Full ISIS Sample Frame
N. Previously Approved Supplementary Documents
O. HPOG-Impact and HPOG-NIE Tracking Letter and Contact Update Form
P. HPOG-Impact and HPOG-NIE Advance Letter
This section provides supporting statements for the collection of information for a National Implementation Evaluation and Impact Study of the Health Profession Opportunity Grants (HPOG) program, funded by the U.S. Department of Health and Human Services (HHS), Administration for Children and Families (ACF). The grants fund programs that provide Temporary Assistance for Needy Families (TANF) recipients, other low-income individuals, and members of Native American tribes with training and support needed to find and keep employment in healthcare occupations and fill the growing demand for skilled healthcare workers. Thirty-two grants were awarded in September 2010 to government agencies, community-based organizations, post-secondary educational institutions, and tribal-affiliated organizations to conduct these activities. Of the 32 HPOG grants, 27 were awarded to agencies serving TANF recipients and other low-income individuals and are relevant to this request.
ACF is implementing a multi-pronged research and evaluation approach for the HPOG program to better understand and assess the activities conducted as part of these grants and their results. The current submission is in support of the HPOG National Implementation Evaluation (NIE) and the HPOG Impact Study. All 27 HPOG grantees serving TANF recipients and other low-income individuals are participating in HPOG-NIE; 20 are participating in HPOG-Impact. Abt Associates and its partner, the Urban Institute, are conducting both evaluations. They build on the previously approved HPOG grantee and participant tracking and management information system called the Performance Reporting System (PRS). There are two Office of Management and Budget (OMB) Information Collection clearance numbers related to this current submission. For the data collection principally for HPOG, under OMB Clearance number 0970-0934, these include:
The Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals, for the Performance Reporting System (PRS) (clearance received September 2011).
The HPOG Impact Study’s baseline data collection instruments (clearance received October 2012).
For the data collection principally for the Innovative Strategies for Increasing Self-Sufficiency (ISIS) project (three HPOG grantees participate in ISIS), under OMB clearance number 0970-0397, these include:
Baseline instruments and implementation interview guides (clearance received November 2011)
The 15-month follow-up survey and related data collection instruments for ISIS (OMB review anticipated in summer 2013).
Other HPOG-related research and evaluation activities include a separate evaluation of the Tribal HPOG grants currently being conducted by NORC at the University of Chicago (OMB clearance number 0970-0395).
ACF and its contractors are engaged in many efforts to coordinate research activities so that each study capitalizes on related work conducted in other projects. By coordinating research efforts, burden is minimized for grantees and for study participants. In addition, comparable data from different, related studies may be combined to enhance the cumulative development of knowledge useful to government policy makers, program operators, and the public. HPOG-NIE and HPOG-Impact will use data from the PRS as well as adapted versions of instruments developed for ISIS, allowing for future analysis of combined data sets. They also make use of data from three ISIS project sites. ISIS is an evaluation of nine career pathways programs (three of which are HPOG grantees) training low-income individuals for various occupations, including healthcare jobs.1
In this document, we request a revision to OMB clearance number 0970-0394 for the first phase of data collection for HPOG-NIE and for follow-on data collection for HPOG-Impact.
ACF seeks approval for the initial HPOG-NIE and follow-on HPOG-Impact data collection activities.
As part of the Affordable Care Act (ACA) of 2010, Congress authorized funds for the HPOG program “to conduct demonstration projects that provide eligible individuals with the opportunity to obtain education and training for occupations in the healthcare field that pay well” (Grant Announcement HHS-2010-ACF-OFA-FX-0126).2 These demonstration projects are intended to address two pervasive and growing problems: the increasing shortfall in the supply of qualified healthcare professionals in the face of expanding demand, and the increasing requirement for a post-secondary education to secure a job with a living wage for families.
Together, HPOG-NIE and HPOG-Impact will collect data to document and demonstrate how effectively grantees implement the HPOG program and to assess how variations in program services affect program outcomes and impacts. As such, HPOG-NIE and HPOG-Impact will fill a void in the sectoral training and career pathways literature both about program effectiveness and about which types of programs or program components are most effective. Few large-scale impact studies of career pathways efforts exist, and none that show the impact of specific program components and models (Werner, Dun Rappaport, et al., 2011).3
H.R. 3590, the ACA requires an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)). The Act further indicates that the evaluation will be used to inform the final report to Congress (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(C)). Examining effects on the demand side of the healthcare sector is a requisite element of the evaluation activities mandated by the ACA in authorizing HPOG. The Act calls for evaluation activities to assess the success of HPOG in “creating opportunities for developing and sustaining, particularly with respect to low-income individuals and other entry-level workers, a health professions workforce that has accessible entry points, that meets high standards for education, training, certification, and professional development, and that provides increased wages and affordable benefits, including healthcare coverage, that are responsive to the workforce’s needs” (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)).
Both HPOG-NIE and HPOG-Impact are guided by the career pathways framework, as shown in the HPOG logic model (Appendix A). The framework puts into practice the assertion that “post-secondary training should be organized as a series of manageable and well-articulated steps accompanied by strong supports and connections to employment” (Fein et al., 2012). These articulated steps provide opportunities for students to advance through successively higher levels of education and training, exiting into employment at multiple possible points. The framework also incorporates customization, supports and employer connections.
Guided by the framework, the goal of HPOG-NIE is to describe and assess the implementation, systems change, and outcomes related to the 27 HPOG grantees focused on TANF recipients and other low‐income individuals. The related goal of HPOG-Impact is to evaluate the effectiveness of approaches used by 20 of the HPOG grantees with regard to improving HPOG participants’ attainment of education, training, employment, and advancement within the healthcare field. Additionally, data from the three HPOG/ISIS grantees may be used in some of the analysis.
HPOG-Impact also is intended to evaluate variation in participant impact that may be attributable to different HPOG program components and models. Twenty of the 27 HPOG grantees serving TANF and low-income individuals will be included in the HPOG impact analysis.4
The HPOG-Impact design includes randomizing program-eligible participants to treatment and control status in all sites. In some (approximately 13) of the sites they will be randomized into two treatment arms (a basic and an enhanced version of the intervention) and a control group. This third experimental arm will create planned variation in some sites that is consistent with natural variation of these program components in other sites, thereby creating an opportunity to learn more both about the effects of program components but also about the methods we use to estimate those effects.
Those in the third arm will participate in one of three planned enhanced HPOG services, in which the basic HPOG program will be augmented by an additional program component. In this subset of grantees, program applicants will be randomly assigned to (1) the “basic” HPOG program, (2) an “enhanced” HPOG program (i.e., the HPOG program plus an enhancement) or (3) a control group that is not offered the opportunity to enroll in HPOG.
Control group members will have access to whatever other programs and services are available in the local community.
In a parallel activity, as mentioned above, the ISIS project has submitted an OMB clearance request (revision of OMB # 0970-0343) for a 15-month follow-up survey for all of the ISIS projects, including the three HPOG grantees that are participating in ISIS. ISIS is also conducting a randomized experiment which will assess similar program impacts. Data from the ISIS baseline survey and follow-up interviews will be included in the HPOG impact analysis.
Additionally, the research team will match participant data collected through HPOG-Impact for both the treatment and control groups to long-term employment and earnings data from ACF’s National Directory of New Hires (NDNH). An agreement with the Office of Child Support Enforcement (OCSE) is in place.
HPOG-NIE will address the following research questions:
1. How are health profession training programs being implemented across the HPOG grantee sites?
2. What changes to the service delivery system are associated with HPOG program implementation?
3. What participant-level outputs (e.g., enrollment, retention, course completions, accreditation/ certification) and outcomes (e.g., job entry, employment retention and advancement, earnings) occur?
4. What key components appear necessary or contribute to the success of these programs?
HPOG-Impact will address the following research questions:
1. What impacts do HPOG programs as a group have on the outcomes of participants and their families?
2. To what extent do these impacts vary by subgroups of interest?
3. Which locally-adopted program components influence average impacts?
4. To what extent does participation in a particular HPOG component (or components) change the impact experienced by individual trainees?
To address these research questions, HPOG-NIE and HPOG-Impact will use a number of data collection instruments. Instruments in the current clearance request include the following:
The HPOG-NIE Sampling Questionnaire for the HPOG surveys (Appendix B)
The HPOG-NIE Follow-Up phone call protocol for the Stakeholder/Network survey (Appendix C)
The HPOG-NIE Grantee survey (Appendix D)
HPOG-Impact Implementation interview guide for partnering employers (Appendix E-1)
HPOG-Impact Implementation interview guide for instructors (Appendix E-2)
HPOG-Impact Implementation interview guide for HPOG program management (Appendix E-3)
HPOG-Impact Implementation interview guide for HPOG program staff (Appendix E-4)
The HPOG-NIE Management and Staff survey (Appendix F).
The HPOG-NIE Stakeholder/Network survey (Appendix G)
The HPOG-NIE Employer survey (Appendix H)
The HPOG-Impact 15-month Participant Follow-Up survey (Appendix I)
The HPOG-Impact 15-month Control Group Member Follow-Up survey (Appendix J)
The HPOG-NIE 15-month Participant Follow-Up survey5 (Appendix I)
These data are not available through any current source.
Study instruments have been cleared in prior clearance requests or are under a current clearance request include the following:
HPOG Performance Reporting System (PRS), a management information system for documenting program activities and accomplishments against program goals and to assist with program management (approved September 2011 under this OMB number);
Supplemental Baseline Questions to the PRS, to be used at participant intake for random assignment into the Impact Study and for analysis for both HPOG-NIE and HPOG-Impact (approved October 2012 under this OMB number); and
ISIS 15-month Follow-Up survey for the three HPOG grantees that are participating in the ISIS project (currently under OMB review under OMB No. 0970-0397).
As part of the HPOG data collection, we anticipate submitting a future additional OMB clearance request for the following:
A 30- or 36-month Follow-Up Participant survey for treatment and control group members in sites participating in HPOG-Impact and among HPOG participants in sites participating only in HPOG-NIE;
Follow-up data collection on children of study participants; and
In-person interviews for HPOG-NIE with HPOG managers and staff to support case studies of selected HPOG grantee programs.
Other extant data sources will be used for the HPOG-NIE and HPOG-Impact studies. These include the following:
National Directory of New Hires (NDNH). These data will provide information about employment and earnings of HPOG participants.
HPOG program management information, including initial applications and ongoing management reports, which we will use as supplemental information in tracking the evaluation of the grant, and information on the local healthcare labor market and needs for occupational training.
Government sources of labor market data, from the U.S. Census Bureau and Bureau of Labor Statistics (BLS), such as County Business Patterns, Local Area Unemployment Statistics (LAUS), Quarterly Workforce Indicators QWI), which will be used to provide a picture of the local labor market.
The HPOG-NIE and HPOG-Impact studies, in conjunction with the related studies under the HPOG research umbrella will increase the knowledge base about the effectiveness of HPOG programs in providing TANF recipients and other low‐income individuals with opportunities for education and training that lead to employment and advancement in the healthcare workforce. We first describe the purpose and approach of the data to be collected primarily for HPOG-NIE, followed by a similar description of data to be collected for HPOG-Impact. However, it should be noted that, for the most part, data collected for one of the two studies will augment and enhance analysis for the other study.
HPOG-NIE involves a set of complementary analyses regarding program implementation and the broader HPOG partner network and system. It includes a description of participant outcomes, an analysis of how participant outcomes relate to participant characteristics and program features, and how, from the perspective of stakeholders and employers, local projects have affected the healthcare labor market. The collection of information through the surveys of various organizations and respondents for HPOG-NIE (including the Grantee survey, the Program Management and Staff survey, the Stakeholder/Network survey, and the Employer survey) will feed into these various analyses. Most of the information collected from HPOG grantees, partners, stakeholders, employers, and respondents from other organizations will be close-ended in nature.
Each of the HPOG grantees includes a unique set of organizations with varying roles operating in different community contexts. HPOG-NIE will rely on representatives from the grantees to develop a grantee-specific sample frame, using the HPOG-NIE Sampling Questionnaire for the HPOG surveys.
In addition, the 15-month Participant Follow-Up survey, which was developed primarily for the HPOG-Impact analysis, will collect information from a cohort of HPOG participants in the four HPOG grantees that are only included in HPOG-NIE. These data will describe longer-term outcomes of participants in the non-Impact Study sites to enhance the descriptive outcome analysis with a 15-month follow-up sample drawn from all 27 TANF and low-income grantees. In addition, the research team will use data collected through the HPOG-Impact site visits to assess and expand upon data collected in the Grantee survey and to provide information about the partnership networks used by grantees.
For HPOG-Impact, baseline data are being collected through the HPOG PRS, including the supplemental baseline questions that were previously cleared under this OMB number. The purposes of these data are several. First, the contact information collected at baseline is necessary to enhance researchers’ ability to locate respondents for follow-up surveys that will measure intervention outcomes. A second purpose is to create a rich dataset for researchers to explore and test hypotheses, including those about the impact of HPOG programs and the relative effectiveness of various components and implementation features of those programs. Other analytic purposes of the baseline data include characterizing the HPOG Impact Study sample, adjusting for chance differences in observable characteristics and thereby increasing precision of impact estimates, identifying subgroups of interest (including program-related subgroups), checking the integrity of random assignment, and adjusting for non-random survey sample attrition. The purpose of the child roster questions is to create a sampling frame for follow-up surveys that collect data about child outcomes.
The 15-Month Participant Follow-Up survey will be conducted approximately 15 months after random assignment. It will collect data on outcomes, including HPOG services received, participation in non-HPOG trainings or services, receipt of degrees or certifications, and employment and earnings outcomes. These data will be used for the HPOG Impact Study to understand treatment and control differentials in participant experiences and outcomes. The research team will identify program features using information from the HPOG-NIE Grantee and Management and Staff surveys as well as information from the HPOG-Impact site visits. Using experimental impact analysis and these data, the research team will estimate the extent to which HPOG program designs lead to differential mean individual outcomes between the treatment and control groups. In assessing the relative impacts of specific program components, the team is combining prospective systematic variation of program models within selected HPOG grantees with natural variation in program models across many HPOG grantees.
The research team will use the HPOG-Impact interview guides included in this clearance package in visits to the 20 HPOG grantees included in HPOG-Impact to interview HPOG program management, staff and service providers. The site visits are important sources of information for both HPOG-NIE and HPOG-Impact. As stated above, the research team will collect most of the data needed to address the HPOG-NIE research goals using largely close-ended-response surveys. While HPOG-Impact will rely principally on the HPOG-NIE close-ended surveys to identify common program features across study sites, the site visits will allow the study team to assess the accuracy and reliability of coded survey responses across grantees and sites. Additionally, the site visits will provide qualitative, nuanced information about how grantees implement otherwise common program elements and will be informed by grantee survey data. Because both evaluations require accurate descriptions of program components and features, the Implementation Study site visits are a critical element of the overall HPOG research strategy.
In the fall of 2013, HPOG-NIE research staff, working with designated site liaisons, will use the HPOG Sampling Questionnaire for the HPOG surveys to identify specific individuals or entities from each of the target respondent groups, including specific respondents from grantee partners, front-line managers and staff, employers, and stakeholders. The research team will build on the list of respondents supplied by the site liaison in the Sampling Questionnaire for the HPOG surveys. In addition to following up and getting contact information as necessary, the team will supplement the original lists to ensure that a broader group of stakeholders beyond those with which the grantee is in direct contact are asked to respond to the survey. To do so, the team will telephone the list of stakeholders identified by the liaison and ask them to name any stakeholders not already identified, and prompt this listing by describing the types and roles and functions that stakeholders may have in the community.
The employers invited to respond to the survey include all of those directly involved with the grantee as a partner in HPOG and, in that capacity, may have been involved in program design, development, and implementation of HPOG. A second group includes those who are not directly involved but are actively hiring, or were contacted about hiring, HPOG graduates. The data collection will target the universe of employers in the first group and draw a purposive sample from the second group.
To reduce burden for employer respondents, employers identify through the information gathering as being both a partner in HPOG and hiring HPOG participants will be asked to fill out one online survey that combines the Stakeholder/Network and Employer survey. Surveys will be fielded in the fall of 2013.
The follow-up survey data collection for HPOG participants and control group members will take place approximately 15 months following random assignment, which begins in late winter and spring of 2013. Consequently, follow-up data collection will begin in summer 2014. The interviews will be conducted primarily by telephone, with field follow-up for those respondents who cannot be reached by telephone.
In the first quarter of 2014, the study team will conduct a round of implementation site visits. Using data collected from the HPOG-NIE Grantee survey, the team will verify information submitted by the grantee and explore any discrepancies between information collected during the site visit and that provided in the grantee survey, as well as document study procedures as they relate to the HPOG Impact Study.
The primary beneficiaries of this planned data collection effort will be ACF, other federal agencies, program operators, other policy makers and researchers, and the healthcare community. ACF will use the information to assess the effects of the HPOG programs on low-income individuals and on the healthcare community. These data will begin to answer ACF's and other policy makers questions about the implementation and impacts of the career pathways programs focused on training staff for the healthcare industry. It will help identify which program components and features appear to result in impacts related to education and credential achievement, employment and earnings, and income and will provide information on the systems change that occur as a result of these program.
Secondary beneficiaries of this data collection will be those in the public policy and social science research community who are interested in further understanding initiatives to promote economic self-sufficiency of individuals and families through comprehensive career pathways programs, particularly as they relate to the healthcare industry. At the conclusion of the HPOG studies, the research team will provide ACF with a restricted-use data set containing individual-level data stripped of all personally identifying information. The restricted-use data will be made available to researchers for approved secondary uses. Ultimately, these data will benefit researchers, policy analysts, and policy makers in a wide range of program areas.
The HPOG-Impact and HPOG-NIE studies involve a relatively large number of separate, but related analyses. These include:
HPOG-NIE: Descriptive Implementation Study;
HPOG-NIE: Systems Change Analysis;
HPOG-NIE: Outcome Study;
HPOG-NIE: Outcome Analysis;
HPOG-Impact: Impact Analysis (including the two- and three-arm experiments);
HPOG-Impact: Analysis of Natural Variation (including connecting it to the planned/experimental variation); and
HPOG-Impact: Qualitative Implementation Study.
Exhibit A-1 describes the target respondents, content, and reason for inclusion (i.e., which analyses will use the information) for each data collection activity.
Exhibit A-1 Item-by-Item Justification of Data Collection Instruments
Data Collection Instrument(s) |
Respondents, Content, and Reason for Inclusion |
ACTIVITY: Development of Sample Frame |
|
1. HPOG–NIE: Sampling Questionnaire for the HPOG surveys (Appendix B) |
Respondents: One to three respondents from each of the 27 HPOG grantees participating in HPOG-NIE (estimated to total 54 over the two-year period)
Content: Requests contact information for individuals who may be potential respondents for the Grantee survey, Management and Staff survey, Stakeholder/Network survey and Employer survey
Used for: Development of the sample frame for the NIE Management and Staff, Stakeholder/Network, and Employer surveys; and identification of appropriate respondents for the Grantee survey |
(Appendix C) |
Respondents: Stakeholders identified by grantees on instrument #1 (estimated to total 162 over the two-year period)
Content: Requests contact information for individuals who may be additional potential respondents for the Stakeholder/Network survey
Used for: Augmentation of the sample frame for the NIE Stakeholder/Network survey |
ACTIVITY: Collection of Data from Grantees |
|
3. HPOG–NIE: Grantee survey (Appendix D) |
Respondents: Staff at each of the 27 TANF and low-income HPOG grantees and their major partners (estimated to total 54 over the two-year period)
Content:
Used for:
|
4. HPOG-Impact: Implementation interview guide for partnering employers (Appendix E-1) |
Respondents: Staff from partner organizations who employ participants after certificate completion at the 20 TANF and low-income HPOG grantee sites (estimated to total 60 over the two-year period)
Content:
Used for:
|
5. HPOG-Impact: Implementation interview guide for instructors (Appendix E-2) |
Respondents: Instructors who teach participants at the 20 TANF and low-income HPOG grantee sites (estimated to total 60 over the two-year period)
Content:
Used for:
|
6. HPOG-Impact: Implementation interview guide for HPOG program management (Appendix E-3) |
Respondents: HPOG management staff in the 20 TANF and low-income HPOG grantee sites (estimated to total 20 over the two-year period)
Content:
Used for:
|
7. HPOG-Impact: Implementation interview guide for HPOG program staff (Appendix E-4) |
Respondents: HPOG staff in the 20 TANF and low-income HPOG grantee sites (estimated to total 80 over the two-year period)
Content:
Used for:
|
8. HPOG–NIE: Management and Staff survey (Appendix F) |
Respondents: HPOG staff in the 27 TANF and low-income HPOG grantee sites (estimated to total 540 over the two-year period)
Content: Survey content will be tied to the respondent’s specific role and include items on responsibilities, opinions on the program and participants, logistics, supports available, and workshop content/structure. Respondent roles include:
Used for:
|
ACTIVITY: Collection of Data from Partners/ Stakeholders |
|
9. HPOG–NIE: Stakeholder/ Network survey (Appendix G) |
Respondents: Staff at organizations partnering with the 27 TANF and low-income HPOG grantee sites and/or stakeholders in the HPOG program (estimated to total 500 over the two-year period)
Content:
Used for:
|
10. HPOG–NIE: Employer survey (Appendix H) |
Respondents: Staff at employers who partner with HPOG in program design, implementation, training, and staff at organizations that employ/may hire HPOG participants from the 27 TANF and low-income HPOG grantee sites (estimated to total 200 over the two-year period)
Content:
Used for:
|
ACTIVITY: Collection of Data from Participants |
|
11. HPOG-Impact: 15-month Participant Follow-Up survey (Appendix I)
|
Respondents: HPOG-Impact study participants assigned to the treatment group (estimated to total 5,600 over the two-year period)
Content:
Used for:
|
12. HPOG-Impact: 15-month Control Group Member Follow-Up survey (Appendix J)
|
Respondents: HPOG-Impact study participants assigned to the control group (estimated to total 2,800 over the two-year period)
Content:
Used for:
|
13. HPOG-NIE: 15-month Participant Follow-Up survey (Appendix I) |
Respondents: HPOG program participants from the four HPOG grantees not participating in HPOG-Impact or ISIS (estimated to total 600 over the two-year period)
Content:
Same as above (#11)
Used for:
|
The HPOG studies will generate a substantial amount of data and will use a combination of data collection methods. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. This evaluation will use improved technology to facilitate the collection of the survey data in standardized and accurate ways that also ensures the protection of the data collected.
The 15-month Participant Follow-Up survey described above will be administered using CATI (computer-assisted telephone interviewing) technology. When the individual cannot be located by telephone, field staff will attempt to find the sample member in person. If the sample member then agrees to take the survey, the field staff will establish contact with the centralized interviewer using a company cell phone. CATI technology reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns work properly, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who did not participate in post-secondary training will be routed past questions only relevant to those who did. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.
Surveys for grantees, management and staff, partners and stakeholders, and employers will be hosted on the Internet via a live secure web-link. This approach is particularly well-suited to the needs of these surveys in that respondents can easily stop and start if they are interrupted, share the link with other respondents, and review and/or modify responses in previous sections. To reduce burden, the research team will pre-select relevant modules for individual respondents, use drop-down response categories so that respondents can quickly select answers from a list, and employ automated skip patterns so respondents are only shown those questions that apply to them.
A.4.1 Surveys and Site Visits
We address this section in three groupings: data collection from the HPOG-NIE surveys that are not of participants; data collection from the 15-month Follow-up surveys; and data collection from the HPOG-Impact site visits.
The purposes of the Grantee, Management and Staff, Stakeholder/Network, and Employer surveys for the HPOG-NIE are to obtain information about HPOG grantee programs, partners and stakeholders, and employer perspectives that is not available through any other source. In addition, the Sampling Questionnaire for the HPOG surveys and the Follow-Up Phone Call Protocol for the Stakeholder/Network survey (#1 and #2 in Exhibit A-1) will be used to collect information on who are the best respondent(s) to the various surveys. Other sources do not currently provide complete information on this. The research team will use existing sources of information, including the PRS, grantee applications, and performance progress reports and will not ask for this same information in the data collection efforts described here. The participant follow-up survey will collect information that is not available from any other sources. This includes information on the control group experiences post-random assignment in the 20 HPOG-Impact sites for which there is no other information as well as information on HPOG participants’ (treatment members in the 20 HPOG-Impact sites and all HPOG participants in the 4 sites that are not participating in HPOG-Impact or in ISIS) post-HPOG experiences.
The information on HPOG training/service receipt and completion available in the PRS for HPOG participants will be used in these studies. Although the team will request some similar information in the 15-month Participant Follow-Up survey, the purpose will be to verify and expand on PRS data. In addition, both studies will use administrative information on wages and employment from the NDNH linked to PRS data that will eliminate the need for gathering as complete an employment history in the survey as might otherwise be necessary. However, these administrative data do not have information on hourly wages, benefits, or other aspects of the job that we will collect in the follow-up survey.
Information collected during the HPOG-Impact site visits will include detailed information on HPOG program design and implementation. Some of the topics addressed will have been addressed in closed-ended question form through the HPOG-NIE Grantee survey. One of the goals of the site visits is to check the reliability of these responses, and the research team will use the results of all the surveys to prepare for these visits. Another goal is to collect additional, more nuanced information in a semi-structured interview format that is not available through a close-ended question survey. This includes more detailed description of challenges to implementation and relevant factors in successful implementation, reasons for choices and actions taken, and further explanation of the perspectives reported in the surveys. To reduce respondent burden in these site visits, site visitors will use all available prior information from the other sources of data collection to that point.
The HPOG and ISIS research teams will work closely to coordinate data collection across both studies. Areas of coordination include:
HPOG-NIE Grantee and Management and Staff surveys will be used by HPOG-Impact to measure program features.
HPOG-Impact site visits will collect information for use in the HPOG-NIE Implementation Study and systems analysis. In addition, ISIS and HPOG teams will conduct site visits jointly to the HPOG programs in ISIS so as to reduce burden on site staff.
Questions and constructs for the 15-Month Participant Follow-Up survey included in this clearance request were based, to the extent feasible on the ISIS follow-up evaluation instruments in order to ensure the alignment of a core group of questions (see Appendix I).
All data collected for the three HPOG sites in ISIS (from surveys modified to include questions specific to HPOG, administrative data, and implementation visits) will be shared with the HPOG research team for inclusion in the HPOG implementation and impact studies.
The primary organizations involved in this study are community colleges, workforce development agencies, employers, and community-based organizations that operate occupational training programs and provide related services. The research team will minimize burden for these entities, including those that could be considered to be small organizations, by requesting only the information required to achieve the study’s objectives and offering them the use of on-line data collection tools so that they can respond to the information request at their convenience. While there will be on-site visits in conjunction with the HPOG Impact Study, the team does not anticipate that any of these will be made to small organizations. In addition, at the time the grants were awarded, ACF informed all grantees of the reporting requirements, and adequate resources have been provided to coordinate the data collection and reporting. There should be no adverse impact for any grantees participating in the study.
All the data that are to be collected associated with this specific burden request are one-time in nature and have not been collected from the specified respondents in any other related OMB clearance request. The data collection effort described in this document is essential to the HPOG-NIE and the HPOG-Impact studies. Less frequent data collection would jeopardize ACF’s ability to conduct these Congressionally mandated studies in time to provide relevant and timely results to shape policy. Collecting data identified in the current request will allow measurement of program features essential for the HPOG-Impact Study and the Descriptive Implementation Study and Systems Change Analysis within the HPOG-NIE, as well as additional outcome measures for both HPOG-Impact and HPOG-NIE.
Data collected through the HPOG-NIE and HPOG-Impact studies are critical to ACF’s comprehensive strategy to evaluate the HPOG demonstration grants. HPOG is a significant policy initiative aimed at training and placing TANF recipients and other low-income individuals in stable healthcare industry occupations with a career path. Many of the HPOG grantees have adopted cutting-edge education and training technologies developed over the past decade to meet the needs of older, non-traditional students with little or no post-secondary educational experience. These relatively new approaches, which generally align with the Career Pathways framework, are largely untested by strong evaluation designs. Together, these studies will develop knowledge about the effectiveness of the new training modules and what works best for whom.
There are no special circumstances for the proposed data collection.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on Thursday, December 20, 2012, Volume 77, Number 245, page 75437, and provided a 60-day period for public comment. A copy of this notice is included as Appendix K. During the notice and comment period, the government received one request for information about the data collection activity. That request was fulfilled. The government received one set of comments from the public. The commenter said the information to be collected seems necessary as a way to measure performance; the accuracy of the time estimates are difficult to gauge but that they do seem reasonable; the information to be collected seems clear and useful, but there is not enough information provided [in the FRN] to allow comment on the quality of the information; and that efforts to minimize the data collection burden could also increase accuracy. The commenter did not request any information.
To address these comments, we took efforts to pretest and edit instruments to ensure burden is kept to a minimum. During internal and external pretesting, all instruments were closely examined to eliminate unnecessary respondent burden and questions deemed to be unnecessary were eliminated. During external pretesting, the research team found that all but one instrument were completed within projections included in Part A of our Supporting Statement. For the 15-month Participant Follow-Up survey, questionnaire pretests led to some item deletions and some consolidation of questions and/or response choices. The changes kept estimated response burden well within the burden estimates in Part A.
Additional steps will be taken to reduce respondent burden where feasible. Prior to fielding the Grantee survey, the research team will tailor the list of occupational trainings offered for each grantee based on information provided in the HPOG Performance Reporting System (PRS). In addition, for grantees in the Impact Study, the research team will pre-populate the Grantee survey with available information. Grantees will be asked to verify the pre-filled information when they complete the survey.
For the Impact Study’s implementation study, site liaisons will review all available data prior to interviews (including PRS data, the Grantee, Management and Staff, Stakeholder/Network, and Employer surveys, and other available information). Interviewers will use information from other sources to inform interview questions such that known information need not be asked again but additional, new data can be obtained. Thus the study team will maximize the information obtained while minimizing burden to respondents.
The HPOG-Impact and HPOG-NIE studies consulted outside experts to inform research designs and analysis plans. These include the following:
Outside Expert |
Affiliation |
Contact Information |
Dr. Larry Hedges |
Professor, Northwestern University |
2006 Sheridan Rd., Evanston, IL 60208 847-491-8899 |
Dr. Carolyn Heinrich |
Professor, University of Texas at Austin |
Lyndon B. Johnson School of Public Affairs, P.O. Box Y, Austin, TX 78713 512-471-3779 |
Ms. Karin Martinson |
Principal Associate, Abt Associates |
4550 Montgomery Ave., Bethesda, MD, 20814 301-347-5726 |
Dr. Jeffrey Smith |
Professor, University of Michigan |
Department of Economics, University of Michigan, 238 Lorch Hall, 611 Tappan St., Ann Arbor, MI 48109 734-764-5359 |
For the HPOG 15-month Participant Follow-Up survey of HPOG participants and control group members in HPOG-Impact sites, we plan to offer participants $30 for completing the follow-up survey, as reflected in the consent form (see Appendix L). This consent form and plan for a participant follow-up survey for the HPOG Impact Study was approved under the previous clearance request for this study. Here, we seek approval for offering the same amount to respondents from the four non HPOG-Impact and non-ISIS grantees. (ISIS sites are also planning to provide the same amount, as indicated in the separate OMB package.)
Additionally, we seek approval to offer $5 for each of the three rounds of participant tracking (see Appendix O). The $5 will be sent to participants after they provide updated contact information. Offering these tokens of appreciation to gain cooperation and solicit participation is a well-established practice in social science research and program evaluation for both small-scale studies and sample surveys. The gesture shows appreciation for voluntary participation in data collection activities and can help ensure high response rates.
Many surveys are designed to offer incentives of varying types with the goal of increasing survey response. Monetary incentives at one or more phases of data collection have become fairly common, including some federally sponsored surveys. Examples include the National Survey on Drug Use and Health (NSDUH, Substance Abuse and Mental Health Services Administration), the National Survey of Family Growth (NSFG, National Center for Health Statistics), the National Health and Nutrition Examination Survey (NHANES, National Center for Health Statistics), the National Survey of Child and Adolescent Well-Being (NSCAW, Administration for Children and Families), and the Early Childhood Longitudinal Study-Birth Cohort (ECLS-B, U.S. Department of Education).
There has been extensive publication about the relative efficacy of different monetary incentives, but several federal agencies have determined $20–$30 to be effective. The U.S. Census Bureau has experimented with and begun offering monetary incentives for several of its longitudinal panel surveys, including the Survey of Income and Program Participation (SIPP) and the Survey of Program Dynamics (SPD). SIPP has conducted several multi-wave incentive studies, most recently with its 2008 panel, comparing results of $10, $20, and $40 incentive amounts to those of a $0 control group. SIPP examined response rate outcomes in various subgroups of interest (e.g., the poverty stratum), use of targeted incentives for non-interview cases, and the impact of base wave incentives on later participation. Overall, $20 incentives increased response rates and improved the conversion rate for non-interview cases. (Creighton et al., 2007). The NSDUH conducted an experiment in which the cost per interview in the $20 incentive group was 5 percent lower than the control group, whereas the $40 incentive group cost was 4 percent lower than the control, due to reduced effort needed in gaining cooperation (Kennet et al., 2005). The NSDUH adopted an intermediate incentive of $30 because the greatest increase in response rate was found in the $20 incentive condition, and the $40 condition obtained a higher variation in per-interview costs. A similar incentive experiment conducted for the NSFG Cycle 5 Pretest examined $0, $20, and $40 incentive amounts. The additional incentive costs were more than offset by savings in interviewer labor and travel costs (Duffer et al., 1994).
Although the 15-month Participant Follow-Up survey itself does not involve collecting individual identification data, the HPOG-NIE and HPOG-Impact data collection will include individual identification data collected through the existing PRS. All HPOG-NIE and HPOG-Impact study participants will complete both the PRS and the Supplemental Baseline Questions added to it for the purpose of the impact evaluation.
The information collected under this data collection will be kept private to the fullest extent provided by law. The information requested under this collection will be private in a manner consistent with 42 U.S.C. 1306, 20 CFR 402, and OMB Circular No. A-130. ACF recognizes that HPOG grantees serve vulnerable populations (per the authorizing legislation), and that grantees must protect those populations from any risks of harm from the research and evaluation activities. Accordingly, as is done when collecting participant data in the PRS, HPOG-NIE and HPOG-Impact will obtain informed consent forms from all study participants. This informed consent will ensure that participants understand the nature of the research and evaluation activities being conducted. The Consent Forms for Participant surveys are included as Appendix L.
As a part of informed consent, grantees will provide the following rationale for data collection and privacy assurances to HPOG participants:
We are conducting this research to see how well various approaches to training for healthcare jobs work. This program and research are funded by the U.S. Department of Health and Human Services, and they may fund other research on this program in the future.
In this program, we will collect some personal information from you, such as your name, date of birth, Social Security number, and your involvement in other programs. The researchers studying the program for the government also need this information. We will keep all of the information about you collected for the program or for the research studies completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
As part of the study, researchers may contact some of you in the future. You may refuse to answer any of their specific questions at any time.
Researchers and program staff using the information collected must take all necessary actions to protect your information and they will pledge their agreement to protect privacy. All Abt Associates employees must sign a data confidentiality pledge on accepting an offer of employment. Any individual allowed access to identifiable data for this project must sign an additional user agreement pledging confidentiality. Urban Institute employees must sign a similar pledge of confidentiality upon employment. Individuals accessing data through the PRS must sign an additional PRS User Agreement that indicates that they will keep those data secure.
The Grantee, Management and Staff, Stakeholder/Network, and Employer surveys are purely voluntary. Respondents will be told that all of their responses during the interview will be kept private, their names will not appear in any written reports, and that responses to the questions are entirely voluntary.
With the exception of some of the 15-month Participant Follow-Up survey, no questions of a sensitive nature will be asked. The survey includes items addressing respondents’ income, welfare receipt, presence of children in the household, and employment barriers such as substance abuse. Some respondents may consider these somewhat personal questions to be sensitive.
Including such items as income, welfare receipt, presence of children in the household, and barriers to employment is necessary to describe the study population and evaluate their moderating effects on program impacts. Furthermore, questions pertaining to personal preferences, motivations and self-efficacy will be especially useful for identifying the pathways that participants follow through multi-faceted programs. This will allow the study team to estimate the impacts of various program models and components, which is the central research question that HPOG-Impact considers. Interview staff will inform respondents that survey participation is voluntary and they may refuse to answer individual items. Study participants will also be reminded that their responses will be kept private, to encourage their candid responses.
The total burden for the instruments already approved (the PRS and Supplemental Baseline Questions) was estimated to be 4,558 hours annually. Estimated burden to continue use of these instruments is 9,244 hours total over two years, or 4,623 hours annually.
Exhibit A-2 presents the reporting burden on study participants completing the instruments included in this data collection request and their total cost. Because some of the data collection instruments will be in the field for longer than one year, burden is annualized and reflected across a two-year period.
We calculated the average hourly wage for each respondent group based on information from the Bureau of Labor Statistics6 or the federal minimum wage. We calculated the average hourly rate7 for each respondent group using the following categories:
Study participant: the minimum hourly wage ($7.25) plus a 40 percent adjustment to account for benefits, or $10.15 per hour.
Community and Social Service Occupations (SOC 21-0000): wage rate of $21.07 plus a 40 percent adjustment for benefits, or $29.49.
Education, Training, and Library Occupations (SOC 25-0000): wage rate of $24.46, plus a 40 percent adjustment for benefits, or $34.24.
Social and Community Service Manager Occupations (SOC 11-9151): wage rate of $30.43, plus a 40 percent adjustment for benefits, or $42.60.
Medical and Health Services Managers (SOC 11-9111): wage rate of $46.17 plus a 40 percent adjustment for benefits, or $64.64.
HR Managers (SOC 11-3121): wage rate of $52.21 plus a 40 percent adjustment for benefits, or $73.09.
When members of a respondent group come from multiple job categories, we took an average across the relevant categories, as noted.
Exhibit A-2: Annual Information Collection Activities and Cost
Instrument |
Total Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total Burden Hours |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Estimated Annual Burden Remaining |
|||||||
PRS |
32 |
4 |
31.2 |
3,994 |
1,997 |
$29.81 |
$59,530.37 |
HPOG-Impact Baseline Survey(s) (Supplemental baseline questions; study sample members) |
10,500 |
1 |
0.25 |
2,625 |
1,313 |
$3.41 |
$4,477.33 |
HPOG-Impact Baseline Survey(s) (Supplemental baseline questions; grantees) |
20 |
525 |
0.25 |
2,625 |
1,313 |
$29.81 |
$39,140.53 |
Estimated Annual Burden New Collection |
|||||||
HPOG–NIE Sampling Questionnaire for the HPOG surveys |
54 |
1 |
2 |
108 |
54 |
$29.49 |
1,592.46 |
HPOG-NIE Follow-Up Phone Call Protocol for the Stakeholder/Network survey |
162 |
1 |
0.17 |
28 |
14 |
$36.058 |
$504.70 |
HPOG–NIE Grantee survey |
54 |
1 |
4 |
216 |
108 |
$31.879 |
3,441.96 |
HPOG-Impact Implementation interview guide for partnering employers |
60 |
1 |
0.50 |
30 |
15 |
$68.8710 |
1,033.05 |
HPOG-Impact Implementation interview guide for instructors |
60 |
1 |
0.75 |
45 |
22 |
$31.87 |
$701.14 |
HPOG-Impact Implementation interview guide for HPOG program management |
20 |
1 |
1.50 |
30 |
15 |
$31.87 |
$478.05 |
HPOG-Impact Implementation interview guide for HPOG program staff |
80 |
1 |
1 |
80 |
40 |
$31.87 |
$1,274.80 |
HPOG–NIE Management and Staff survey |
540 |
1 |
0.5 |
270 |
135 |
$31.87 |
$4,302.45 |
HPOG–NIE Stakeholder/Network survey |
500 |
1 |
0.5 |
250 |
125 |
$36.0511 |
$4,506.25 |
HPOG–NIE Employer survey |
200 |
1 |
0.5 |
100 |
50 |
$68.8712 |
$3,443.50 |
HPOG-Impact 15-month Participant Follow-Up survey |
5,600 |
1 |
0.7 |
3,920 |
1,960 |
$10.15 |
$19,894.00 |
HPOG-Impact 15-month Control Group Member Follow-Up survey |
2,800 |
1 |
0.6 |
1,680 |
840 |
$10.15 |
$8,526.00 |
HPOG–NIE 15-month Participant Follow-Up survey |
600 |
1 |
0.7 |
420 |
210 |
$10.15 |
$2,131.50 |
Total Burden Hours: New Collection |
7,177 |
3,588 |
|
$51,829.86 |
|||
TOTAL Burden Hours (Remaining + New) |
16,421 |
8,211 |
|
$154,978.09 |
|||
Exhibit A-2 displays total burden. The total burden for already approved information collection and the new request is 16,421 hours, or 8,211 hours per year over two years.
This data collection effort involves no recordkeeping or reporting costs for respondents other than those described in Exhibit A-2 above.
The total cost for these data collection activities will be $7,670,821. Annual costs to the Federal government will be $3,835,411 for the proposed data collection. Total annual cost of the information collection including previously approved information collection is $10,781,711. This includes the cost of developing and pretesting data collection instruments and tools, administering the surveys (including the Sampling Questionnaire for the HPOG surveys, the Follow-Up Phone Call Protocol for the Stakeholder/Network survey, the Grantee survey, the Management and Staff survey, the Stakeholder/Network survey, the Employer survey and the 15-month Participant Follow-Up surveys), and collecting implementation research data.
This is a request to add additional questions to an already approved collection.
HPOG-Impact and HPOG-NIE have complementary analysis plans. NIE reports will focus on the structure of the programs designed and implemented by the grantees as well as on immediate outputs of the programs in simple terms such as numbers of graduates. Impact Study reports will focus on evaluating the overall effectiveness of the grant program, as well as evaluating the relative efficacy of different program designs and studying linkages between specific program features and personal participation on the one side to student outcomes on the other side.
While additional reports may be produced, current plans include the following reports:
An interim NIE report on baseline descriptive outcomes. This report will be completed by August 2013. It will profile participants in terms of data collected in the Performance Reporting System (PRS).
An NIE report on descriptive implementation and outcomes. This report will contain two major sections. This report will be completed by November 2014. The first section will be a comprehensive description of the HPOG program as designed and implemented across 27 grantees and their sites. The section will cover, for example: program context, including local labor market characteristics, program operations, resources, and costs, and individual level outcomes or HPOG participants. The report will also include updates to the interim report on baseline descriptive outcomes using additional available data from the PRS.
An NIE systems change and network analysis report. This report will discuss changes to the service delivery system associated with program implementation. In addition, the Systems Change Analysis will describe and analyze the institutional and stakeholder network in which the HPOG program operates. This report will be completed by December 2014.
An NIE final report. This report will use the 15-month Participant Follow-Up survey data from participants to give a more complete understanding of the conditions of employment. This report will be completed by September 2017. Some of the outcomes that can be studied with participant survey data (but could not be studied with data from the PRS or NDNH) include: post-program employment and earnings in a health job, wages, benefits, further career training and career advancement.
An Impact evaluation final report. This report will focus on how average outputs (including education and training experiences) and outcomes (including, credential/ certificate/degree attainment, employment, earnings/wages, job benefits and other characteristics) differ between the randomized groups, differences that—when statistically significant—are attributable to the HPOG program since no other systematic differences exist. This report will be completed by June 2016. The data will be examined in multiple ways to address the multiple research questions, including:
All treatment group members compared to all control group members;
Comparison of treatment and control group members for individuals in specific demographic categories such as women, high school dropouts, and non-native English speakers;
Comparison of treatment and control group members whose HPOG programs provide certain intervention components; and
Comparisons of treatment and control group members with equivalent baseline characteristics that, in the treatment group, are associated with participation in a particular intervention component.
A key feature of the Impact Study is the exploitation of the substantial cross-site variation in program design and implementation, both via planned and natural variation, to address the program component-focused research questions. This report will focus on program experiences and the treatment-control contrast, early program impacts on outcomes such as credential attainment and impacts on intermediate outcomes such as employment and earnings as well as job quality.
Upon completion, each of the final reports will go through ACF’s thorough review process. As part of the review process, ACF will ensure each report is 508 compliant for dissemination on their website.
Exhibit A-3 presents an overview of the project schedule for information collection. It also identifies publications associated with each major data collection activity.
Exhibit A-3: Overview of Project Data Collection Schedule
Data Collection Activity |
Timing |
Associated Publications |
Baseline data collection for HPOG-Impact |
March 2013–November 2014.13 |
Baseline Data Collection Report (February 2015) |
Survey sample frames |
Upon OMB Approval–September 2013 |
NA |
Surveys of HPOG grantees, management/staff, stakeholders, and employers |
October through December 2013 |
Descriptive Implementation and Outcome Report (November 2014) Systems Change and Network Analysis Report (December 2014) |
Implementation interviews with HPOG personnel |
January–March 2014 |
Implementation Study Site Visits Report (July 2014) |
15-month Participant Follow-Up survey |
March 2014–November 2015 |
Impact Evaluation Final Report (June 2016) Final NIE Report (September 2017) |
All instruments created for HPOG-NIE and HPOG-Impact will display the OMB approval number and the expiration date for OMB approval.
No exceptions are necessary for this information collection.
Creighton, K., King, K., and Martin, E. (2007). The Use of Monetary Incentives in Census Bureau Longitudinal Surveys. Research Report Series, Survey Methodology, #2007-2, January. Washington, DC: U.S. Census Bureau.
Duffer, A., Lessler, J., Weeks, M., and Mosher, M. (1994). Effects of Incentive Payments on Response Rates and Field Costs in a Pretest of a National CAPI Survey. Paper presented at the Annual Meeting of the American Statistical Association, Toronto.
Fein, D., Judkins, D., Martinson, K., Rolston, H., Gardiner, K., Bell, S., and Barnow, B. (2012). Innovative Strategies for Increasing Self-Sufficiency Evaluation Design Report. Unpublished draft manuscript. Bethesda, MD: Abt Associates Inc.
Hendra, R., Dillman, K., Hamilton, G., Lundquist, E., Martinson, K., and Wavelet, M. (2010). The Employment Retention and Advancement Project: How Effective Are Different Approaches Aiming to Increase Employment Retention and Advancement? Final Impacts for Twelve Models. New York: MDRC.
Kennet, J. and Gfroerer, J. (Eds.) (2005). Evaluating and improving methods used in the National Survey on Drug Use and Health (DHHS Publication No. SMA 05-4044, Methodology Series M-5). Rockville, MD: Substance Abuse and Mental Health Services Administration, Office of Applied Studies.
Maguire, S., Freely, J., Clymer, C., Conway, M., and Schwartz, D. (2010, July). Tuning In to Local Labor Markets: Findings from the Sectoral Employment Impact Study. Philadelphia: Public/Private Ventures.
Roder, A., and Elliott, M. (2011, April). A promising start: Year Up’s initial impacts on low-income young adults’ careers. New York, NY: Economic Mobility Corporation.
Werner, A., Dun Rappaport, C., and Lewis, J. (2011). Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals, Draft literature review: Career Pathway Programs. Cambridge, MA: Abt Associates Inc.
1 The ISIS consent form used by the three ISIS sites that are also HPOG grantees includes language indicating that data will be shared with HPOG researchers.
2 Authority for the HPOG demonstrations is included in the Patient Protection and Affordable Care Act (ACA), Public Law 111-148, 124 Stat. 119, March 23 2010, sect. 5507 (a), “Demonstration Projects to Provide Low-Income Individuals with Opportunities for Education, Training, and Career Advancement to Address Health Professions Workforce Needs,” adding sect. 2008(a) to the Social Security Act, 42, U.S.C., 1397g(a).
3 Public Private Ventures’ Sectoral Employment Impact Study (Maguire et al., 2010) is an impact evaluation of sectoral employment programs; A Promising Start: Year Up’s Initial Impacts on Low-Income Young Adults’ Careers (Roder and Elliott, 2011) is a small-scale random assignment impact study of a sectoral employment effort that does not target healthcare. The impact evaluation of the national Employment Retention and Advancement (ERA) Project (Hendra et al, 2010) is another recent impact study of a workforce development program, but it is not specifically focused on career pathways or healthcare.
4 Three HPOG grantees being evaluated under ISIS (though data collection is being coordinated, and data from the HPOG/ISIS grantees may be used in the HPOG Impact Study) and four HPOG grantees who are engaged in independent research projects with a University partner are not included in the HPOG Impact Study.
5 HPOG-NIE and HPOG-Impact will use the same instrument for the 15-month Participant Follow-up survey.
6 http://www.bls.gov/oes/current/oes_nat.htm
7 Assuming 2080 FTE hours worked.
8 Source: Bureau of Labor Statistics, National Compensation Survey, 2011: Combined average hourly wage of Community and Social Service Occupations and Social and Community Service Manager Occupations
9 Source: Bureau of Labor Statistics, National Compensation Survey, 2011: Combined average hourly wage across Education, Training, and Library Occupations and Community and Social Service Occupations was used for the grantee survey, survey of HPOG program management and staff, and interviews with HPOG personnel.
10 Source: Bureau of Labor Statistics, National Compensation Survey, 2011: Combined average hourly wage of Medical and Health Services Manager and HR Manager
11 Source: Bureau of Labor Statistics, National Compensation Survey, 2011: Combined average hourly wage of Community and Social Service Occupations and Social and Community Service Manager Occupations.
12 Source: Bureau of Labor Statistics, National Compensation Survey, 2011: Combined average hourly wage of Medical and Health Services Manager and HR Manager.
13 Currently operating under previous OMB clearance (approved September 2011 and October 2012)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |