Pathways for Advancing Careers and Education (PACE) – Third Follow-up Data Collection
OMB Information Collection Request
OMB No. 0970-0397
Supporting Statement
Part A
June 2017
Submitted by:
Nicole Constance
Office of Planning,
Research
and Evaluation
Administration for
Children
and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Supporting Statement for OMB Clearance Request – Part A: Justification
Table of Contents
A.1 Necessity for the Data Collection 1
A.1.2 Legal or Administrative Requirements that Necessitate the Collection 2
A.2 Purpose of Survey and Data Collection Procedures 2
A.2.1 Overview of Purpose and Approach 2
A.2.4 Universe of Data Collection Efforts 4
A.2.5 Data Collection Process 7
A.2.6 Instrument Item-by-Item Justification 7
A.3 Improved Information Technology to Reduce Burden 8
A.4 Efforts to Identify Duplication 8
A.4.2 Coordination and Streamlining of Study Efforts 9
A.5 Involvement of Small Organizations 9
A.6 Consequences of Less Frequent Data Collection 9
A.8 Federal Register Notice and Efforts to Consult Outside the Agency 9
A.9 Incentives for Respondents 10
A.10 Privacy of Respondents 14
A.10.1 Data Privacy Protections 14
A.12 Estimation of Information Collection Burden 15
A.12.1 Data Collection Already Approved 15
A.12.2 Current Information Collection Request 15
A.12.3 Total Burden Hour Request 16
A.13 Cost Burden to Respondents or Record Keepers 16
A.14 Estimate of Cost to the Federal Government 16
A.16 Plan and Time Schedule for Information Collection, Tabulation and Publication 16
A.16.2 Time Schedule and Publications 17
A.17 Reasons Not to Display OMB Expiration Date 18
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 18
Appendices:
Appendix A: Career Pathways Framework
Appendix B: PACE Program Summaries
Appendix C: PACE 72-Month Follow-up Survey
Appendix D: PACE 72-Month Follow-up Survey – Sources
Appendix E: PACE Federal Register Notice
Appendix F: PACE Previously Approved Participant Contact Update Form
Appendix G: Contact Update Letters
G1: PACE Previously Approved Contact Update Letter for 36-Month Survey
G2: PACE 72-Month Follow-up Contact Update Letter
Appendix H: PACE Previously Approved Informed Consent Form
Appendix I: PACE 72-Month Participant Newsletter
Appendix J: Survey Flyer
J1: PACE Previously Approved Flyer for 36-Month Survey
J2: PACE 72-Month Flyer
Appendix K: Email Text
K1: PACE Previously Approved Email Text
K2: PACE 72-Month Email Text
Appendix L: Survey Advance Letters
L1: PACE Previously Approved 36-Month Survey Advance letter
L2: PACE 72-Month Advance Letter
Appendix M: Previously Approved PACE 36-month Follow-up Survey
Appendix N: Contact Update Call Script
Appendix O: Comments from Federal Register Notice and ACF Response
O1: Comments from Federal Register Notice
O2: ACF Response to Comments
In this document, we provide justification for the next set of data collection activities for the Pathways for Advancing Careers and Education (PACE) evaluation sponsored by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS).1
OMB approval was received in November of 2011 for PACE baseline data collection, in August of 2013 for the PACE 15-month data collection, and in December of 2014 for the PACE 36-month data collection (OMB No. 0970-0397).
This submission seeks OMB approval for one follow-up data collection instrument:
A 72-Month Follow-up Survey
PACE is one project within the broader portfolio of research that OPRE is using to assess the success of career pathways programs and models. This strategy also includes a multi-pronged research and evaluation approach for the Health Profession Opportunity Grants (HPOG) program to better understand and assess its activities and results. In order to maximize learning across the portfolio, development of the PACE and HPOG 1.0 Impact baseline and follow up surveys is being coordinated, and the majority of the data elements collected in these surveys are similar. Four data collection efforts for HPOG 1.0 Impact have been approved under OMB clearance number 0970-0394, and a fifth (new) request is being submitted at the same time as this request. The HPOG 1.0 Impact and PACE research teams coordinated on development of the 72-month survey.
ACF seeks approval for the 72-month follow-up data collection activities described in this request in order to support a study conducted for it by Abt Associates (Abt).
ACF conceived of the PACE project as a test of promising interventions for improving the economic prospects of low-income individuals and families. After extensive outreach to the program and policy community (conducted under OMB clearance No. 0970-0343), ACF determined that the focus of PACE would be programs that fit into the career pathways framework. Appendix A is an exhibit of the career pathways framework and the theory of change used to guide the PACE evaluation.
The PACE evaluation will assess a range of promising postsecondary career pathways programs that promote the improvement of education, employment, and self-sufficiency outcomes for economically disadvantaged adults. The major goal of PACE is to assess the effectiveness of a group of these programs in increasing 1) the receipt of educational credentials, 2) employment and earnings, and 3) self-sufficiency and other measures of adult well-being. ACF believes the development of rigorous evidence on these matters will be of great use to both policymakers and program administrators. The PACE 72-Month Follow-Up Study will help to address key questions regarding the long-term effects of the PACE programs. This data collection extends these studies with three additional years of follow-up on the impact of the PACE programs.
This six-year time horizon provides an exciting opportunity to discover important long-term program impacts, including:
Are any early employment or earnings impacts sustained?
Do some programs have late-blooming effects on earnings?
Are these programs helping low-income individuals obtain better jobs (e.g., jobs with insurance, paid leave, more steady work hours, or enhanced responsibilities)?
Are treatment group members more likely than controls to follow a career pathway, demonstrated by returning to school or moving up an occupational career ladder?
Are these programs helping families become self-sufficient?
This effort will help answer these questions overall and for various subgroups of particular policy interest, such as TANF recipients, single parents, or people with certain baseline levels of education or work histories.
There are no legal or administrative requirements that necessitate this data collection. ACF is undertaking the collection as part of its ongoing effort to improve the economic well-being of the low-income population.
The PACE project is an evaluation of promising programs and policies for improving employment and self-sufficiency outcomes for low-income, low-skilled adults. The PACE study is utilizing an experimental design in nine programs to assess the impact of promising interventions on education and training credential attainment, employment, earnings, and general well-being and will also include an implementation study and a cost-benefit study.
The 15-Month Participant Follow-Up Survey approved under this OMB Clearance Number collected data on outcomes, including PACE services received, participation in non-PACE trainings or services, receipt of degrees or certifications, and employment and earnings outcomes. These data are used to understand treatment and control differentials in the experiences and early outcomes of study participants.
The 36-Month Participant Follow-Up survey, approved under this OMB Clearance number and currently underway, will allow for an understanding of the experiences and intermediate outcomes of study participants in both the treatment and control groups. Using experimental impact analysis and these data, the research team will estimate the extent to which PACE programs lead to differential mean individual outcomes between the treatment and control groups.
The purpose of this third follow-up survey is to measure approximately 72 months after random assignment participant outcomes and program impacts on employment progression, educational attainment, current employment conditions, student debt, financial well-being, and other life circumstances. These data will be used for the PACE impact study. The instrument can be found in Appendix C.
The primary beneficiaries of this planned data collection effort will be ACF, other federal agencies, program operators, and low-income individuals. ACF will use the information to assess the effects of the PACE programs for low-income individuals. These data will help to answer ACF's questions about impacts of the postsecondary career pathways programs in all study domains: education and credential achievement, employment and earnings, income, and adult and child well-being. Similarly, the Departments of Labor and Education have expressed a strong interest in the PACE study in particular and career pathways program effectiveness in general. The results of the PACE study could inform programmatic and funding decisions for all three agencies. Organizations (e.g., community colleges, workforce development agencies, community-based organizations) that are operating or creating career pathways programs will use the study information to refine or design programs for their target populations. Finally, low-income individuals will benefit from this information to the extent that it demonstrates the cost-effectiveness of career pathways programming and contributes to a body of evidence to inform program and policy design and investments in the approach.
Secondary beneficiaries of this data collection will be those in the public policy and social science research community who are interested in further understanding initiatives to promote economic self-sufficiency of individuals and families through comprehensive career pathways programs. At the conclusion of the PACE study, the research team will provide ACF with a restricted-use data set containing individual level data stripped of all personally identifying information. The restricted-use data will be made available to researchers for approved secondary uses.
Ultimately, these data will benefit researchers, policy analysts, and policy makers in a wide range of program areas. The effects of postsecondary career pathways programs on the well-being of low-income individuals and families could manifest themselves in many dimensions and could be relevant to an array of other public programs. This project offers the first opportunity to obtain reliable measures of these effects. The long-term indirect benefits of this research are therefore likely to be substantial.
Overall, the PACE evaluation seeks to address the following research questions:
Implementation—What services are provided under each intervention? What are the characteristics of the populations served? How are services implemented? What are the issues and challenges associated with implementing and operating the service packages and policy approaches studied? How do services available to the treatment group compare to the services available to the control group? How does the take-up and utilization of services by the treatment group compare to the take-up and utilization in the control group?
Implementation data will provide a fuller understanding of the conditions surrounding these career pathway programs and the contexts in which they operate. This information also will allow researchers to assess the quality of the implementation of these programs—assessments that will be important to the interpretation of program impact results.
Impact—What are the net impacts of career pathway programs on educational outcomes (program completion and attainment of credentials and degrees), economic outcomes (earnings, employment levels, and wage progression), and adult well-being?
The impact study will use baseline data, 15- month follow-up survey, 36-month follow-up survey, and 72-month follow-up survey data (for which this package seeks approval) to address impact study research questions. Additionally, data on study participants’ wages will be collected from the National Directory of New Hires (NDNH). This is expected to reduce the burden on study participants by negating the need to ask detailed earnings questions on the follow-up surveys. Additionally, administrative records are not subject to recall error or non-response. The research team will continue to use the National Student Clearinghouse (NSC) to gather information about college persistence and degree completion for study participants. NSC is currently the only data source which tracks postsecondary student enrollment across states. NSC data help to reduce the need to ask detailed questions on postsecondary school enrollment, thereby reducing the respondent burden.
Data collected for PACE will provide a rich body of information from which to answer these key research questions. For example, the research team will be able to say with confidence whether a program improved credential receipt, household income, and career growth at the time of the 72-month follow-up. This package seeks OMB approval the 72-month follow-up survey.
Cost effectiveness—What are the costs of career pathway programs in the study? Do the estimated benefits of providing services outweigh the costs of these programs?
The project will address the cost-effectiveness of programs through comparison of net economic benefits with net program costs in the cost-benefit study. The bulk of cost data will come from programs’ existing administrative records and other administrative data. Survey data will be used to account for receipt of non-wage income.
PACE study sites targeted low-income adults who were interested in occupational skills training. The sites conducted random assignment of individuals to one of two groups: a treatment group that was offered the innovative career pathways interventions, or a control group that was able to access a set of “business-as-usual” services, or any other services available in the community except the PACE services. The sample size in eight of the nine sites ranges from 500 to 1,220 with the sample equally distributed between the two research groups. The ninth site (Year-Up) has an overall sample of 2,542 across eight sub-sites, with 1,670 in the treatment group and 872 in the control group. Appendix B provides summaries of the nine PACE programs.
In 2011, 2013 and 2014, the PACE project obtained OMB clearance for the baseline data, 15-month follow-up data collection, and 36-month follow-up data collection (OMB No. 0970-0397). The following instruments were approved under those clearances:
Basic Information Form (BIF) for participants collected general demographic and contact information. The BIF was administered during intake prior to random assignment by an intake staff person or self-administered on a paper form. The BIF was modified to collect information at baseline regarding the sample members’ children (as applicable) to establish a sampling frame for future follow-up activities that estimate the effects of the programs on the children of those in the study who are parents. (Approved November 2011, with modifications approved in August 2013)
Self-Administered Questionnaire (SAQ) for participants collected more sensitive and personal information, including several psycho-social items designed and validated by the testing firm ACT, Inc. The SAQ was also completed during intake prior to random assignment and was self-administered on a paper form. (Approved November 2011)
First round Interview Guides for interviews with program staff that were used to collect information from site staff personnel. The project team interviewed program administrators and staff at PACE sites and other organizations that partnered with PACE sites to deliver services. The first round interview guides were used for interviews conducted during the pilot and early full implementation stages. (Approved November 2011)
15-month Follow-up Survey. This follow-up survey collected information from study participants 15 months following the date of random assignment. The survey was administered by telephone using specially trained interviewers and captured information on outcome measures for treatment and control group members in several domains including education and training, employment and income, and life circumstances. Field follow-up was used to contact participants who could not be reached after multiple phone attempts. (Approved August 2013)
Second Round Interview Guides for Program Leadership/Managers, Instructional Staff, Case Managers/Advisors, and Partners. Interview topic guides for the implementation study were used during a second round of site visits to each program to collect information from PACE program staff and other organizations involved in the delivery of services. The interview guides collected data to describe the programs as implemented, including core components, management and staffing, and contextual factors. In addition to describing the interventions, this information will help the research team interpret impact results. (Approved August 2013)
Online Surveys of Case Managers/Advisors, Managers/Supervisors and Instructional Staff. Online staff surveys were administered at each of the nine PACE programs. The case manager/advisor survey focused on the issues covered with students (personal, academic, career planning, employment, financial), the amount of time spent with students, staff development activities, and how closely student progress and completion were monitored. The managers/supervisor survey inquired about staff background, the nature of assistance provided to program participants, and staff development and morale. The instructor survey elicited quantitative data about class size, the extent to which basic skills were integrated with training instruction, time spent on different instructional modes, instructor backgrounds, staff development activities, staff autonomy, and morale. (Approved August 2013)
In-depth Study Participant Interviews. In-depth interview guides were used to collect information from a sample of study participants from each site at two points in time, as well as for a brief interim telephone check-in. In the seven programs that were single sites (i.e., no sub-sites), the team interviewed 10 treatment and five control group members. In the two programs with sub-sites, the team interviewed 10 treatment and five control group members in each of the three sub-sites in the I-BEST program and 10 treatment and five control group members in four of the eight sub-sites in the Year Up program (NCR, Boston, Bay Area, and Chicago). This information will be used to gain a more comprehensive understanding of treatment and control members’ experiences with the services. (Approved August 2013)
36-month Follow-up Survey. This follow-up survey collects information from study participants approximately 36 months following random assignment. The follow-up survey is being administered by telephone, with in-person follow-up for those not completed by telephone, using specially trained interviewers. The survey captures information on intermediate outcome measures for treatment and control group members in several domains including education and training, employment and income, and life circumstances. (Approved December 2014) (See Appendices F, G1, J1, K1, L1 and M for all previously approved materials related to the ongoing 36-month Follow-up Survey.)
Data collection for all previously approved instruments is now complete, with the exception of the 36-month follow-up survey, which is underway. The current submission seeks approval for the 72-month follow-up data collection instrument developed for the PACE evaluation. The new follow-up survey, when approved, will collect information from a sample of study participants approximately 72 months following random assignment. Local interviewers will administer all interviews using CAPI (computer assisted personal interviewing) software on encrypted laptops or tablets. The survey will capture information on longer-term outcome measures for treatment and control group members in several domains including education and training, employment progression, income and financial well-being, and life circumstances. (Appendix C) Many of the questions to be asked at 72 months were approved for the 36-month survey and most other items have been asked in other OMB-approved studies. A summary of the survey items and sources is provided in Appendix D.
The data collection instruments described above require direct interaction with study participants or program staff. The PACE study also draws heavily upon administrative data from other sources. Those sources provide data not captured in the survey and they impose no respondent burden. The other data sources include:
Government administrative records: These records include Unemployment Insurance (UI) and federal wage records. OPRE has established an agreement with the HHS Administration for Children and Families’ Office of Child Support Enforcement to utilize the UI and wage records from the National Directory of New Hires (NDNH).
National Student Clearinghouse: The National Student Clearinghouse (NSC) data system includes more than 3,600 participating public and private institutions that collectively represent approximately 98 percent of higher education enrollments nationwide. The PACE study will use NSC data for information on college persistence and degree completion.
Program records: The project team collected data on outcomes from the programs operating the PACE interventions. These data will be used for the implementation study and for the impact study in sites where program records were available. Illustrative outcomes include measures of basic academic skills, services received, and credits and credentials earned. The information included in these records differs from site to site based on the information collected by each program's management information system(s). Where a community college operated the studied program, the study team was able to get reasonably comparable data on both treatment and control group members; but for the most part, program data were limited to the treatment group. This did not impose burden on programs because they collected these data for their own use.
The 36-month follow-up survey began in February 2015 and will continue through May 2018. The follow-up survey data collection will take place approximately 72 months following random assignment, which began in the first program in November 2011. Therefore, the 72-month follow-up data collection will start in late 2017 following OMB approval. The last cohort to be released includes participants that went through random assignment in May 2014. Hence, the data collection is expected to conclude in early 2021. If needed, a request for an extension will be submitted to OMB prior to the expiration of OMB #0970-0397.
Exhibit A-1 describes the target respondents, content, and reason for data collection for the new data collection activity. For more information about previously approved instruments, see previously approved information collection requests under OMB # 0970-0397. A copy of the currently active 36 month survey instrument that was previously approved is provided in Appendix M. The new 72-month survey is in Appendix C and a summary of the survey questions and sources for the new 72-month instrument is in Appendix D. All other 72-month survey support materials are provided in Appendices F, G2, I, J2, K2, L2, and N.
Exhibit A-1: Justification of Data Collection Instruments
Data Collection Activity |
Data Collection Instrument(s) |
Respondents, Content, and Reason for Collection |
Study Participant Follow-up Survey |
72-Month Follow-up Survey
(Appendix C) |
Respondents: Overall expected completed interviews from 4,400 study participants from the PACE programs selected for the 72-month Follow-up Survey (see Part B for description of program selection)
Content:
Reason: This follow-up period of 72 months will provide a longer-term look at the sustainability of early and interim education and employment impacts, overall adult well-being, and an opportunity to measure child outcomes after they have aged six years. |
The PACE evaluation will generate a substantial amount of data and will use a combination of data collection methods. For each data collection activity, the study team has selected the form of technology that enables the collection of valid and reliable information in an efficient way while minimizing burden. This evaluation will use improved technology to facilitate the collection of the survey data in standardized and accurate ways that also ensures the protection of the data collected.
The follow-up survey will be administered using CAPI for all interviews. CAPI technology reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns work properly, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who did not participate in postsecondary training will be routed past questions only relevant to those who did. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.
The purpose of the follow-up survey for the PACE evaluation is to obtain current information on the status and wellbeing of individuals in the PACE evaluation study sample 72 months after study enrollment. Information about these respondents' educational achievement, economic well-being, and job skills development and progression and overall well-being are not available through any other source, nor is information about family composition, student debt, or child outcomes. The evaluation will utilize administrative data (e.g., wage records) in conjunction with survey data to avoid duplication of reporting.
The research team will also avoid duplication in this study by use of a study-specific database, maintained by Abt, which links all the data collected at baseline and the prior follow-up survey efforts with subsequent information gathered from administrative sources. This eliminates the need to ask about personal characteristics or background factors for known household members on follow-up surveys.
Of the nine sites included in PACE, three were programs that received Health Profession Opportunity Grants (HPOG) administered by ACF and a fourth is a sub-grantee to an HPOG-funded program. These programs were funded under the first round of HPOG grant funding (2010-2015). ACF is also funding implementation and impact evaluations of the first round of HPOG program (HPOG 1.0). The PACE and HPOG research teams worked closely to coordinate data collection in the four programs that are part of both studies. Areas of coordination include:
Development of the 72-month follow-up survey included in this clearance request. The teams worked in close collaboration to develop the follow up instrument. The HPOG 1.0 Impact evaluation is also submitting an OMB clearance package at this time under OMB #0970-0394.
Data sharing. All data collected for the HPOG 1.0 sites in PACE will be shared with the HPOG research team for inclusion in the HPOG implementation and impact studies.
The primary organizations involved in this study are community colleges, workforce development agencies, and community-based organizations that operate occupational training programs. Burden was minimized for these entities by requesting the minimum information required to achieve the study’s objectives. On-site interviews with program staff covered topics on which the study team was unable to collect sufficient information by other means. Interviews with program staff are completed.
The data collection effort described in this document is essential to the PACE evaluation. If data were collected less frequently, it would jeopardize ACF’s ability to conduct the impact analyses. Delays in the administration of the follow-up survey run an inherent risk that the respondent will have trouble recalling the details about the questions posed, with the achievement of key milestone events potentially missed as study participants move through their training and education and employment progressions.
The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public, General Information Collection Guidelines). There are no circumstances that require deviation from these guidelines.
In accordance with the Paperwork Reduction Act of 1995, the Administration for Children and Families (ACF) at the Department of Health and Human Services published a notice in the Federal Register January 13, 2017, Vol. 82, No. 9, page 4341. The document number is FR Doc. 2017–00583. A copy of the notice is shown in Appendix E. During the notice and comment period, the government received one request for information about the data collection activity.
The commenter expressed his support of this data collection effort, but suggested we: (1) look into using the Census Bureau’s Longitudinal Employer-Household Dynamics (LEHD) database as a replacement for NDNH data to obtain UI wage records; (2) review the National Center for Educations Statistics’ Adult Training and Education Survey (ATES) to see if it might inform the design of the education and credential section; and (3) asked that we inform the Workforce Information Advisory Council (WIAC) on PACE research findings when they are available. Early in the study, we had considered LEHD data because it has UI wage data matched to other administrative and survey data, which could have reduced the survey length. We chose to use NDNH wage data instead because the available LEHD data do not meet the study’s needs: they do not cover the study’s follow-up period and the matched data does not cover the entire study population. As documented in Appendix D, however, we did use some of the education and credential questions from the ATES survey to improve the 72-month survey. Appendix O contains the full set of comments received (Appendix O1) and ACF responses (Appendix O2).
To ensure the length of the instrument is within the burden estimate, we took efforts to pretest with fewer than 10 people and edit the instruments to keep burden to a minimum. During internal pretesting, all instruments were closely examined to eliminate unnecessary respondent burden and questions deemed unnecessary were eliminated.
Monetary incentives show study participants that the study team appreciates the time they spend participating in study information collection activities. Although published evidence of the effectiveness of incentives in reducing nonresponse bias appears to be nearly nonexistent, it is well established that incentives strongly reduce attrition in panel studies.2 In accordance with OMB guidelines, the team took several factors into consideration when determining whether or not to use incentives.3 Specifically, the team took into account data quality issues, efforts to reduce non-response bias, the complexity of the study design and panel retention over a 72-month period, and prior use of incentives for this study population.
In longitudinal studies such as PACE, panel retention during the follow-up period is critical to minimizing the risk of nonresponse bias and to achieving a sufficient sample size for analysis. Although low response rates do not necessarily lead to nonresponse bias and it is at least theoretically possible to worsen nonresponse bias by employing some techniques to boost response rates (Groves, 2006), most statisticians and econometricians involved in the design and analysis of randomized field trials of social programs agree that it is generally desirable to obtain a response rate as close to 80 percent as possible in all arms of the trial (Deke and Chiang, 2016). The work of Deke and Chiang underlies the influential guidelines of the What Works Clearinghouse (WWC). Under those guidelines, the evidential quality rating of an evaluation is downgraded if the difference in response rates between arms exceeds a certain tolerance (e.g., 4 percentage points when the overall response rate is 80 percent).
Mindful of these risks and the solid empirical base of research demonstrating that incentives do increase response rates, OPRE and OMB authorized incentives for the prior rounds of data collection at 15 and 36 months (OMB control number 0970-0397). In an effort to maximize response rates, the team periodically requests that participants update their contact information, through a contact update form (See Supporting Statement Part B, for more information on the contact update process and Appendices F, G1 and G2 for the previously approved contact update form, previously approved contact update letter and new contact update letter respectively). OMB authorized incentives for both completion of the survey and for the contact updates. At 15 months, the incentive was $30 for completing the follow-up survey interview and $5 for providing updated contact information in advance of the scheduled interview time. With these incentives, PACE achieved response rates for the treatment groups varying from 72.4 percent to 90.8 percent across the nine sites. Response rates on the control groups were generally lower, varying from 68.5 percent to 79.4 percent. Overall, PACE achieved a 77.2 percent response rate and experienced a differential response rate of 5.1 percentage points. Site-specific gaps varied from -1.0 to +13.1 percentage points.
Given these response rates and gaps, at 36 months the conditional incentive for completing the main interview was increased to $40, the $5 incentive for updating contact information was changed to a $2 prepayment included in the request for a contact update, and a prepayment of $5 was added to the advance letter package to remind research subjects of the study and note that a legitimate interviewer would be calling them shortly to learn about their experiences since study enrollment.
In most longitudinal studies, response rates decline over follow-up rounds. The team has tried to minimize this expected decline and ensure a high response rate with a low treatment-control differential through the continued use of the participant contact update forms and the provisions of tokens of appreciation. Through these tools the team hopes to address three goals:
Overcome participant mobility—over a long follow-up period, many study participants relocate multiple times, making it difficult to find them and update their contact information or complete a follow-up interview;
Reduce survey data collection costs— the quicker interviewers can locate the respondent and complete an interview, the lower the costs per completed survey, and
Maintain participant engagement in a complex panel study—the ability to keep participants engaged in the research study six years after enrollment is crucial to understanding long-term outcomes.
All study participants received periodic requests to update their contact information using the previously approved contact update form, in the time between the 15- and 36-month follow-up surveys. Participants receive tokens of appreciation for providing the updated contact information leading up to the 36-month follow-up survey. The study team will continue to send these update letters to all participants who have not reached the 36th month after random assignment yet. The team will use the same contact update form, with revised letters, prior to the 72-month survey and, pending approval from OMB the team will again provide tokens of appreciation to participants that complete. The purpose of these forms is to ensure we have up-to-date contact information so that we can ensure all participants have equal likelihood of being reached when it is time to conduct their interviews.
The participant contact update form does not collect any data for analytic use, but these updates are crucial to ensuring that the contact information in the sample database is as up to date as possible during the follow-up period. Although the team is not aware of any true experiments on the effects of requests for contact updates, the team does have strong circumstantial evidence of their effectiveness in raising response rates in the follow-up survey. This evidence arises in particular from the first follow-up survey for the PACE study, where no requests were sent to early cohorts, but requests were sent to subsequent cohorts. While the final response rate differential did not differ substantially between the earlier and later cohorts, the earlier cohorts had to be worked much longer to achieve completion targets. Those early cohorts were worked about 10 to 12 months to completion, about 4-6 months longer than the later cohorts. The research team knows from experience on the 15-month and 36-month follow-up surveys that this sample is difficult to locate. The contact update forms help to ensure that the contact information in our records is accurate. Accurate contact information allows field interviewers to complete more cases by telephone, which reduces the average hours per complete and costs associated with in-person locating and interviewing efforts. Keeping the hours per complete low for most of the sample, allows additional time and resources to put toward efforts to interview the hardest to locate participants. Updated contact information also helps to keep the overall field period short—allowing cases to be interviewed closer to the 72-month random assignment anniversary date and ensuring the utility of the survey to explore factors influencing progression for outcomes such as cumulative credits earned and attainment of credentials, which are very sensitive to the lag between randomization and interview.
Abt Associates is currently about mid-way through data collection for the 36-month follow-up. Perhaps due to the increased incentives among other efforts (such as the periodic contact updates and an established rapport with interviewers from the 15-month data collection) currently the average response rate is only one point lower than it was for the 15-month follow-up.
The 72 month follow-up survey data collection begins three years later, so the team developed a protocol to help retain the panel and keep study subjects engaged in the study. The contact update form (Appendix F) will be used again, but the form will be supplemented with a participant newsletter (Appendix I) and a contact update check-in call (Appendix N). The participant newsletter and the check-in call address the challenges of the participant mobility and participant disengagement from the study over time. The first contact update request will be sent in the traditional way, accompanied by a letter to participants 12 months prior to the start of data collection.4
The team will resend the contact update form eight months prior to the start of survey data collection, accompanied by a participant newsletter. This newsletter is intended to help show participants the importance of their continued study participation.
The team will also look to strengthen participant engagement through a brief contact update call. Interviewers will conduct the contact update check-in call four months prior to the start of survey data collection, and collect updated contact information. The check-in call will also remind participants about their role in the study, alert them of upcoming data collection efforts, and allow them a chance to ask any questions.
Since two and a half to three years have elapsed since the participants were last contacted by the research team, the use of incentives will also aid in the re-engagement effort as our prior experience with this study population shows that it does respond positively to incentive payments.
Three factors helped to determine the incentive amounts for each survey:
Respondent burden, both at the time of the interview and over the life of the study;
Costs associated with participating in the interview at that time; and
Other studies of comparable populations and burden.
The PACE 72-month follow-up study survey sample is a subset of the original sample (up to 6,000 participants or roughly 66 percent of the total PACE sample). Given the need to measure and report site level impacts, further emphasizes the importance of minimizing attrition. Study participants have been contacted every three to four months over the first three years of the follow-up period and asked to participate in our contact update efforts or a follow-up survey. They will also be asked to participate in the re-engagement efforts leading up to the 72-month follow-up study. This is a long period of time for participants to remain engaged in the research.
To fully utilize the data collected through the 72-month follow-up survey, we believe it is necessary to take every possible step to minimize panel attrition over the study follow-up period. This minimal attrition rate is the core justification for an incentive system for the follow-up interview. The need to maintain the panel is further complicated by the high mobility rates experienced in prior rounds of data collection with this study population. Tokens of appreciation help to secure the cooperation of the individual over the duration of the study period and reduce the potential for individuals to fail to complete the survey. This is particularly important for this round of data collection as only a subsample of participants will be selected for interview.
Despite these re-engagement efforts, given a target response rate of 74 percent for the 72-month follow-up, and based on the incentive amounts approved for previous rounds of data collection with the potential respondents, we feel it would be wise to further increase incentives, as well as to slightly restructure them. The incentive amounts proposed (subject to OMB approval) for the 72-month survey and contact update responses are as follows:
$5 token of appreciation for responding to the contact update letters;
$10 token of appreciation for completing the contact update call;
$45 token of appreciation for completing the survey.
These tokens of appreciation are provided to help offset any potential expenses incurred by the participant such as cell phone minutes for those completed by telephone, and childcare, or transportation costs for those completed in-person. The proposed amounts take into consideration the incentive structure approved leading up to the 15- and 36-month survey efforts as well as changes to the overall sample retention approach. The amount requested for completion of the 72-month survey is a modest increase for a survey completed about three years after our last contact with participants. The contact update call is new to this study population, while it captures similar information to the contact update form, it does require additional burden on the participants to complete this by telephone. For that reason, we propose increasing the amount provided to $10 rather than $5. We propose returning to the previously approved $5 conditional incentive for those who return their contact update form. These incentive rates and proposed increases are comparable to what was offered and previously approved in prior rounds of PACE data collection. These rates are also similar to what was previously approved under the first round of HPOG grants (0970-0394) and is comparable to what is proposed for the 72-month follow-up effort for that study.
The information collected under this data collection will be kept private to the fullest extent provided by law. The information requested under this collection will be private in a manner consistent with the previously approved informed consent document participants signed at the time of enrollment (see Appendix H).
The study team has established rigorous data security and privacy provisions. First, all data users are aware of and trained on their responsibilities to protect participants’ personal information, including the limitations on uses and disclosures of data. (Each study team member who works with data signs an Individual Investigator/Confidentiality Agreement, which outlines the individual’s responsibilities in complying with the standards and requirements for protecting data). The research databases are designed to limit access to authorized users with levels of access commensurate with each person’s role on the project. The web server hosting the database is maintained in a secure facility with power back up, network redundancy, and system monitoring. In addition, daily back up of the server is maintained at the data center and an off-site location. The database and website are password protected, and access is provided only after user authentication.
The PACE Participation Agreement (see Appendix H), completed at the time of random assignment, ensures a commitment to keeping personal information private. This assurance is also made to all respondents as part of the introduction to the follow-up surveys. For both survey data and corresponding administrative data on sample members, computer security will be maintained by individual passwords and folder permissions which limit access to files to only those project staff members who require access to these files.
Each study subject provided her/his (1) last and first name; (2) Social Security number; and (3) date of birth at the time of enrollment. The last and first names and Social Security Numbers are needed by evaluators to obtain accurate administrative data on individuals’ quarterly earnings and receipt of cash and noncash public benefits—data needed to measure key impacts. These personal identifiers also help with matching to administrative data on educational records. Administrative data matches will only be done for those participants who provided informed consent. Individuals were told how the data would be used and how the data would be securely stored.
The follow-up survey includes one question each about overall physical health (E5), emotional health (E4), and whether substance use is a barrier to work or family responsibilities (E3c), items that some respondents may consider sensitive. The literature provides ample support for including these items as barriers to education and employment. These items will help to describe the study population and evaluate mediating effects on program impacts. Program staff will remind study members during the interviewing process that they may refuse to answer individual items. Study members will also be reminded that their responses will be kept private to encourage their candid responses.
Previously Approved and Completed Data Collection Burden
The total burden for the baseline and 15-month instruments already approved was estimated to be 13,839 hours. Baseline and 15-month data collection is complete.
Interim Outcome (36-month Follow-up Survey) Data Collection Already Approved
The total burden for the 36-month follow-up survey already approved was estimated to be 7,386 hours, or 2,462 hours annually. Administration of the survey continues and the total remaining burden is 3,075 hours, or 1,025 hours annually over the three years of this current request.
Exhibit A-2 shows the estimated burden for the new instruments and instruments which have previously been approved but have burden remaining. It shows the average time, in hours, that study participants are estimated to spend completing each data collection instrument.
The average hourly wage used to put a dollar value on burden hours was calculated for each respondent group based on information from the Bureau of Labor Statistics5 and the federal minimum wage. The mean hourly rate6 for each respondent group was calculated as follows:
Study participant: the federal minimum hourly wage ($7.25) plus a 40 percent adjustment to account for fringe benefits in employment, or $10.15 per hour.
Exhibit A-2: Total Data Collection Burden
[This information collection request is for a three-year period.]
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of responses per respondent |
Ave. burden hours per response |
Annual burden hours |
Ave. Hourly Wage |
Total Annual Cost |
Previously Approved Instruments Remaining Burden |
|||||||
36-Month Follow-up Survey |
3,075 |
1,025 |
1 |
1 |
1,025 |
$10.15 |
$10,404 |
Current Request for Approval |
|||||||
72-Month Follow-up Survey |
4,400 |
1,467 |
1 |
.75 |
1,100 |
$10.15 |
$11,165 |
Total |
2,125 |
|
$21,569 |
||||
The figures in Exhibit A-2 imply that the total burden for already approved, but continuing, information collection (the 36-month follow-up survey) and the new request (72-month follow-up survey) is 6,375 hours, or 2,125 hours per year over three years. The annual burden is equivalent to $21,569 based on respondents’ estimated hourly compensation of $10.15, or a total of $64,707 over three years.
This data collection effort involves no costs for respondents other than those described in item A.12 above.
The total cost for the data collection activities (and all related analyses and reports) under this current request will be $9,577,603. This amount includes costs for new data collection activities under this request and the remaining costs from previously approved collections still in progress. Annual costs to the Federal government will be $3,192,534 for the proposed data collection under this OMB clearance number (0970-0397).
This evaluation involves new data collection that increases the public reporting burden under this OMB number. Additionally, some data collection under this OMB number has been completed. Section A-2 details the burden figures.
The evaluation contractor, Abt Associates, and its subcontractors will collect, analyze, tabulate and report the data collected for the PACE evaluation to ACF.
The PACE data collection activities will support the following major deliverables in addition to those already produced under the same OMB number:
Nine Site-specific 36-month impact studies. Each interim report will describe the program impact on key indicators, including education enrollment and credential attainment, earnings and employment, and family well-being. Interim impact reports will be drafted on a rolling basis based on a schedule determined by the timing of the 36-month surveys. The first report is projected to be finalized by March 2018 and the last by April 2019. In addition, six site-specific cost-benefit analyses reports, using data from the 36-month survey, will be developed and submitted in 2019.
Nine Site-specific 72-month impact studies. For each PACE program selected for the 72-month survey, the final report will describe the program impact on key indicators, including career pathways-relevant training; earnings, job quality and career-track employment; family economic self-sufficient and well-being. In addition, site-specific cost-benefit reports, using data from the 72-month survey, will also be developed. For PACE programs not selected for the 72-month survey (See Part B for site selection plans), impact estimates for quarterly enrollment and number of terms enrolled will be analyzed based on administrative data. Impact reports for the 72-month follow-up period will be drafted on a rolling basis based on a schedule determined by the timing of the 72-month surveys (which is based on the timing of random assignment at each site). All reports will be finalized by September 2021.
Upon completion, each final report undergoes ACF’s thorough review process. As part of the review process, ACF will ensure each report is 508 compliant for dissemination on their website. All published reports are on the Career Pathways website:
http://www.career-pathways.org/recently-published/ or
https://www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers-and-education
Exhibit A-3 presents an overview of the project schedule for all information collection under OMB #0970-0397. It also identifies deliverables associated with each major data collection activity.
Exhibit A-3 Overview of Project Data Collection Schedule
Data Collection Activity |
Timing |
Associated Publications |
|
Concluded |
Site-specific implementation reports, interim impact reports (forthcoming 2017)
See Fein, David (2016). Risk Factors for College Success: Insights from Adults in Nine Career Pathways Programs. |
|
Concluded |
Site-specific implementation reports, interim impact reports (forthcoming 2017) |
|
Concluded |
Site-specific implementation reports, interim impact reports, cost-benefit report (forthcoming 2017) |
|
Concluded |
Site-specific implementation reports |
|
Concluded |
Site-specific implementation reports (forthcoming 2017)
See PACE Profiles for each site in References See Fein, David (2016). Scaling Up to Close the Opportunity Divide for Low-Income Youth: A Case Study of the Year Up Program. |
|
Concluded |
Site-specific implementation reports (Forthcoming 2017) See multiple Seefeldt et. al., 2016 reports in References
|
|
Currently operating under OMB # 0970-0397 |
Site-specific, interim and final impact reports (Forthcoming between December 2017 and April 2019.) |
|
Beginning in 2017 upon OMB approval |
Site-specific and summary final impact reports (Forthcoming between December 2020 and September 2021). |
All data collection instruments created for the PACE evaluation will display the OMB approval number and expiration date.
This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
Deke, J. and Chiang, H. (2016). The WWC attritional standard: Sensitivity to assumptions and opportunities for refining and adapting to new contexts. Evaluation Review. [epub ahead of print]
Fein, David (2016). Risk Factors for College Success: Insights from Adults in Nine Career Pathways Programs. OPRE Report #2016- 36, Washington, D.C.: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Fein, David (2016). Scaling Up to Close the Opportunity Divide for Low-Income Youth: A Case Study of the Year Up Program. OPRE Report #2016-55, Washington, D.C.: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Groves, R. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646-675.
Judkins, D., Morganstein, D., Zador, P., Piesse, A., Barrett, B., Mukhopadhyay, P. (2007). Variable Selection and Raking in Propensity Scoring. Statistics in Medicine, 26, 1022-1033.
Lynn, P. (ed.) (2009). Methodology of Longitudinal Surveys. Chichester: John Wiley & Sons.
Seefeldt, Kristin, Engstrom, Whitney and Karen Gardiner. (2016). Programmatic and Other Supports Accessed by Career Pathways Participants. OPRE Report #2016-29, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Seefeldt, Kristin, Engstrom, Whitney and Karen Gardiner. (2016). Finances, Family, Materials, and Time: Career Pathways Participants’ Perceived Challenges. OPRE Report #2016-28, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Seefeldt, Kristin, Engstrom, Whitney and Karen Gardiner. (2016). Nothing Can Stop Me: Career Pathways Participants’ Motivations and Thoughts on Success. OPRE Report #2016-27, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
PACE PROFILES
Engstrom, Whitney, Fein, David, and Gardiner, Karen. (2014). Pathways for Advancing Careers and Education Career Pathways Program Profile: Year Up. OPRE Report #2014-51, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Copson, Elizabeth, Gardiner, Karen and Howard Rolston. (2014). Pathways for Advancing Careers and Education Career Pathways Program Profile: Valley Initiative for Development and Advancement (VIDA). OPRE Report # 2014-21, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glosser, Asaph, Hamadyk, Jill, Gardiner, Karen and Mike Fishman. (2014). Pathways for Advancing Careers and Education Career Pathways Program Profile: Washington’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges. OPRE Report #2014-38, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Hamadyk, Jill and Karen Gardiner. (2014). Pathways for Advancing Careers and Education Career Pathways Program Profile: Des Moines Area Community College WTA Connect Program. OPRE Report #2014-39, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Copson, Elizabeth, Martinson, Karin and Karen Gardiner. (2014). Pathways for Advancing Careers and Education Career Pathways Program Profile: Instituto del Progreso Latino’s Carreras en Salud Program. OPRE Report # 2014-20, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Glosser, Asaph, Hamadyk, Jill and Jessica Wille. (2014). Pathways for Advancing Careers and Education Career Pathways Program Profile: Workforce Development Council of Seattle-King County’s Health Careers for All Program. OPRE Report # 2014-20, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Engstrom, Whitney, Gardiner, Karen and Olenka Mitukiewicz. (2013). Pathways for Advancing Careers and Education Career Pathways Program Profile: Pima Community College Pathways to Healthcare HPOG Program. OPRE Report # 2013-31, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Elkin, Sam, Farrell, Mary and Jessica Willie. (2013). Pathways for Advancing Careers and Education Career Pathways Program Profile: San Diego Workforce Partnership’s Bridge to Employment in the Healthcare Industry. OPRE Report # 2013-20, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Cook, Rachel, Gardiner, Karen and Karin Martinson. (2013). Pathways for Advancing Careers and Education Career Pathways Program Profile: Madison Area Technical College Patient Care Pathway Program. OPRE Report # 2013-20, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
1 From the project inception in 2007 through October 2014 the project was called Innovative Strategies for Increasing Self-Sufficiency (ISIS).
2 See Chapter 12 of Lynn (2009), in particular, section 12.5 that reviews the effects of incentives in several prominent longitudinal studies.
3 See page 69, questions 75 and 76, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf
4 Given the timing of OMB review and approval and the planned start of data collection, the early cohorts will have a compressed contact update request process.
5 http://www.bls.gov/oes/current/oes_nat.htm
6 Assuming 2,080 FTE hours worked.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Missy Robinson |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |