SUPPORTING STATEMENT FOR THE PAPERWORK REDUCTION ACT OF 1995
SUBMISSION FOR DATA COLLECTION FOR THE EVALUATION OF
THE YOUNG PARENTS DEMONSTRATION, PART A
The U.S. Department of Labor’s Employment and Training Administration (ETA) is seeking Office of Management and Budget (OMB) approval to: (1) collect participant data from organizations that received grants under the Young Parents Demonstration Program (YPDP); (2) conduct semi-structured interviews with key administrators and staff in the demonstration projects to document the structure and implementation of the demonstration intervention; and (3) conduct follow-up surveys of YPDP participants at 18 and 36 months after random assignment.
The YPDP is designed to provide educational and occupational skills training that fosters family economic self-sufficiency to young parents (both mothers and fathers) and expectant parents ages 16-24, including, as applicable, those in high-risk categories such as victims of child abuse, children of incarcerated parents, court-involved youth, youth at risk of court involvement, homeless and runaway youth, Native American youth, migrant youth, youth in or aging out of foster care, and youth with disabilities.1 The YPDP grantees are required to develop a mentoring model, which includes an intensive professional staff mentoring specifically for education, employment, and training and specifically for pregnant and parenting teens and young parents. They are to implement this intervention as an additional level of services above and beyond the existing services they currently provide that are specifically intended to increase an individual’s education, job training and employment.
Individuals enrolling in the program have a 50/50 chance of receiving this additional level of services. Those individuals assigned to the treatment group will receive the additional services, while individuals assigned to the control group will receive the existing services offered by the grantee. To evaluate the YPDP mentoring interventions, education, employment, and other outcomes of the two groups will be compared over time.
The Urban Institute (a nonprofit, non-partisan research organization based in Washington, DC), Capital Research Corporation, Inc. (a for-profit, small business evaluation research firm located in Arlington, VA), and Abt Associates (a for-profit, research firm located in Boston, MA and Bethesda, MD), referred to from here on as the “evaluation team,” are collaborating on designing and conducting process/implementation and net impact evaluations of YPDP. Among the evaluation activities being conducted by these three firms are the following: (1) provision of assistance to each YPDP site to implement rigorous random assignment procedures to ensure fidelity in the assignment of YPDP participants to treatment and control groups; (2) development and implementation of a web-based Participant Tracking System (PTS) to collect valid, reliable, and comparable participant-level data across all the grantee sites; (3) ongoing monitoring of enrollment in YPDP and of random assignment procedures; (4) documentation of the structure and implementation of the interventions based on site visits to each YPDP grantee; and (5) administration of follow-up surveys at 18 months and 36 months after enrollment to obtain employment, earnings, and educational outcomes, as well as participant views on services received and their effectiveness.
Thus, the evaluation involves three types of data collection activities that are the principal focus of this OMB supporting document, which are highlighted below.
Participant Tracking System (PTS). For their pre-existing programs, all grantees maintain data on individual participants they serve in a variety of formats such as computerized management information systems, Microsoft Excel spreadsheets, and paper files. To ensure rigorous process and impact evaluations, it is necessary to establish a uniform system of data collection by standardizing the participant data on demographic characteristics, services received, and outcomes across the sites. This web-based PTS is just such a uniform system which also helps execute the random assignment procedures. A detailed description of the PTS is provided in Attachment A.
Field-based Implementation Site Visits. The second research task involves conducting site visits to the grantee sites aimed at documenting the program environment, the start-up, existing and bump-up services, participant flow through random assignment and program services, program costs, and grantee perspectives on implementation challenges and intervention effects. A two-person team will spend an average of two days on each grantee site approximately 12 months after the start of the random assignment process. During this visit, up to nine key administrators and staff will be interviewed using a semi-structured interview guide which is presented in Attachment B.
Telephone Follow-up Survey. Using Computer Assisted Telephone Interviewing (CATI), a survey team will collect data from both treatment and control group members on the type and intensity of services received, including the type and strength of the mentoring relationship; perspectives on their participation in the program; and various outcomes relating to employment, education, and parenting. The survey data will be used as part of the impact study to estimate short-term net impacts of YPDP participation – for example, estimating differences between the control and treatment groups in employment and earnings outcomes at 18 and 36 months after random assignment. We anticipate conducting telephone surveys with all YPDP participants at the sites. The enrollment goal for YPDP across the four sites is 1,633 participants (i.e., a total of slightly over 800 treatment and 800 control group participants). With an 80 percent response rate, this survey effort will yield a total of 1,306 YPD participants completing 18- and 36-month follow-up interviews. Details of the follow-up surveys can be found in section A.2 of this statement. In addition, Attachments C and D provide copies of the survey instruments to be used in conducting the 18- and 36-month telephone surveys.
1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The FY 2010 Omnibus Appropriations Bill required that the Secretary establish a demonstration project to address the employment and training needs of young parents under the Pilots, Demonstration and Research provisions (section 171) of the Workforce Investment Act of 1998, 2 grants were awarded in June 2011 to organizations in Los Angeles, CA; Asheville, NC; Birmingham, AL; and Worcester, MA. These organizations will implement intensive professional staff mentoring specifically for education, employment, and training and specifically for pregnant and parenting teens and young parents. Additionally, the Department has commissioned Capital Research Corporation and the Urban Institute to perform an evaluation of the demonstration under section 172 of WIA. The evaluation will use a rigorous experimental design featuring random assignment of YPDP participants into treatment and control groups.
A web-based Participant Tracking System (PTS) has been developed to accomplish random assignment in an efficient and confidential manner and allow ongoing quality control activities necessary to maintain the integrity of the random assignment process. The participant data items collected through PTS at the time of enrollment are necessary to execute the random assignment process. In this process, there is a 50 percent chance of each YPDP participant being assigned to either the treatment (bump-up) or the control group. The PTS has been designed to allow the grantees to collect data on participant characteristics, services received, and a select group of short-term participant outcomes, such as whether the participant is employed and hourly wages received at 6, 12, and 18 months after random assignment. The PTS also contains participant contact information (e.g., address, telephone, and e-mail information, as well as alternative contact information for three individuals who know the participant) to facilitate service provision and the administration of the follow-up surveys.
The site visits are an integral part of the process/implementation study for documenting the environment in which each of the four programs will operate, the flow of YPDP participants through random assignment and throughout the program services, the types of interventions available for both the control and treatment groups such as existing and bump-up services, and other programmatic characteristics. An in-depth understanding of the structure of each program and services being offered for treatment and control group members is necessary to establish the extent and nature of differences between the services received by treatment and control group members. Understanding the specifics of how each of the demonstration sites operates and the services delivered is essential to determining if any differences in outcomes between the treatment and control groups that exist are potentially associated with differences in interventions received. It is also imperative to understand the details of program operations and services provided should the Department of Labor (or other state or local agencies) seek to replicate part or all of the YPDP interventions in the future.
The follow-up telephone surveys at 18 and 36 months after random assignment are needed to collect detailed information about employment, earnings, educational, parenting, and other outcomes across YPDP participants. Such outcome data will be used to statistically analyze net impacts of YPDP participation. In addition, this survey effort will provide the ability to gather more qualitative views of participants (in a comparable way) concerning their experiences with program participation.
2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
The follow-up surveys of YPDP treatment and control group members at 18 months and 36 months after random assignment, the Participant Tracking System, and the field-based implementation site visits to YPD projects are each a new collection. The information to be collected will be used to understand and analyze the impacts of the program overall and for different bump-up services. The key research questions being addressed are shown in the table below (see Table 1), which displays study questions and the principal data sources to be used to address each of the study. The results of the study will be used by policy makers and youth program providers in the Department of Labor to improve services to young parents. Since each of these is a new collection, there has been no use of the information yet.
Participant Tracking System (PTS). The four YPDP sites will use the PTS to collect participant data necessary at the time of intake to randomly assign YPDP participants into treatment and control groups. This system will ensure that each YPDP site collects valid and comparable data required for random assignment. Once data are entered into the PTS, the system has an algorithm that will automatically and randomly assign each entered individual into the control or treatment groups. Hence, the PTS will ensure that rigorous and unbiased procedures are used to assign individuals according to the experimental design being used to evaluate net impacts of YPDP. YPDP sites will also use the PTS to: (1) record contact information to facilitate locating participants throughout their involvement in YPDP (e.g., to facilitate case management, mentoring, and service delivery) and to facilitate locating participants for the planned follow-up surveys; (2) systematically collect data on services received by YPDP participants during their involvement in the demonstration and specifically on mentoring services; and (3) to collect and record comparable employment and educational outcome data on participants at 6 months, 18 months, and 36 months after random assignment. The PTS will provide valid and comparable data across the four YPDP sites to be used by the evaluation team to analyze participant characteristics, services received, and short-term outcomes. YPDP site staff will enter data into the PTS on a secure website; participant information that might identify individuals will be immediately encrypted and complies with all provisions of the Urban Institute’s Institutional Review Board (IRB). (See Attachment A for a description of the PTS, the informed consent, mentoring log and, instructions to grantee staff about using the PTS approved by the Urban Institute’s IRB.)
TABLE 1: KEY EVALUATION QUESTIONS AND DATA SOURCES
KEY EVALUATION QUESTIONS |
PTS |
Site Visits/ Process Study |
Follow-up Surveys |
Quarterly Earnings |
Question #1: To what extent are there statistically significant differences in employment, education, parenting, and other outcomes for the bump-up (treatment) and control groups? |
|
|
|
|
Question #2: How do net impacts on key outcomes vary across YPD sites for the treatment and control groups? |
|
|
|
|
Question #3: If it is possible to pool samples across YPD sites and sample size is adequate for subgroup analyses, how do net impacts on key outcomes (for the treatment and control groups) vary for specific subpopulations of the youth served (e.g., race/ethnicity, gender, educational attainment, and other demographic factors)? |
|
|
|
|
Question #4: Based on the results of the follow-up surveys and other evaluation activities (e.g., the process evaluation), what are the potential reasons for variation in net impacts for treatment and control groups? |
|
|
|
|
Question #5: What types of services/assistance do participants receive, how satisfied are they with the services received (e.g., the strength and intensity of their relationship with their mentor), and to what extent is achievement of employment, education, and other outcomes associated with the types of services received? |
|
|
|
|
Question #6: Based on the results of the follow-up surveys (and cost data collected through the process evaluation), what are the most cost-effective strategies for delivery of services to improve employment, education, and other outcomes for YPD participants? Are there specific strategies that should be adopted to meet the needs of specific subpopulations of youth? Are there some strategies or subgroups for which the intervention appears ineffective? Are there ways that the intervention can be improved based on these findings? |
|
|
|
|
Field-based Implementation Site Visits. A rigorous evaluation requires clear and specific documentations of the existing and bump-up program interventions received by control group and treatment group members in each of the grantee sites. To develop such documentation, a team of two experienced evaluators will visit each site and follow a specific, detailed field visit protocol and interview guide. The site visit data collection will follow a standard protocol for conducting semi-structured interviews with selected staff and administrators. Each individual staff or administrator interview will be conducted in a private office or room on-site following established procedures for maintaining strict individual privacy. Notes from the interview will be handwritten or entered onto a laptop computer. After each visit, the field notes will be stripped of any identifying information to guard against any violations of privacy provisions. Notes will be stored in a secure computer or file cabinet at the Urban Institute or its partners that can only be accessed by the evaluation team. (See Attachment B for the interview instrument.)
Telephone Follow-up Surveys. The follow-up surveys of YPDP treatment and control group members at 18 months and 36 months after random assignment will provide standardized and statistically valid data for analyses of impacts of YPDP services on such outcomes as employment and earnings, and educational attainment. The survey will provide data from randomly assigned treatment and control group members on the following topics (note: see Attachments C and D for copies of the 18- and 36-month follow-up survey instruments and specific questions to be asked under each of these topics):
Service Receipt
Mentoring Services
Educational Attainment Since Random Assignment
Employment and Earnings
Receipt of Cash Assistance
Receipt of Food Stamps and Other Assistance
Family Composition/Change
Relationship/Engagement with Children
Food Security
Housing and Housing Security
Family Income/Contact Information
The surveys will be one of several key data sources to support the analysis of the net impact on participants of the YPDP intervention. Survey data will be carefully integrated with data from the PTS, qualitative findings from site visits conducted to each of the YPDP sites, and other administrative data (e.g., Unemployment Insurance wage record data) to report on outcomes, net impacts, and cost effectiveness of YPDP). All results and materials developed from the analyses of this data collection effort are intended to reach multiple audiences, including:
ETA and DOL staff,
Policymakers at the state and federal levels of government looking to design programs and services to be responsive to the needs of parenting youth;
Employment and training/workforce development groups and associations; and
Community colleges, technical colleges, workforce investment agencies and organizations, other similar training providers, and human service agencies providing a range of services for parenting youth.
The final reports produced under the process/implementation and impact analysis components of the YPDP evaluation (which will include the survey results) will be posted on the ETA website. The surveys, and the reports they inform, will help ETA better understand whether and how employment, training, education, and parenting services can and should be implemented at the state and local levels to assist young parents and expectant youth. The survey results will help ETA, states, and local workforce investment areas to identify potential problems/challenges to effectively serving parenting youth. These results will also inform possible changes to policy, strategies, and service delivery to enhance program performance and cost-effectiveness of services provided to young and expectant parents through a wide variety of local service providers including One-Stop Career Centers, workforce investment boards, community-based organizations, and educational/training institutions.
3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.
Participant Tracking System (PTS). The PTS is a web-based information system used for gathering background information on participants and services received, as well as executing random assignment for the evaluation. The use of this technology minimizes the burden on grantee staff to enter data and execute random assignment. The YPDP grantee staff enter data into the PTS and use the system to randomly assign participants into the treatment and control groups. In addition to being trained by the evaluation contractor on how to collect and enter data into the PTS, each site has been provided with a user manual fully documenting PTS data collection and entry procedures. In addition, a web-based training module on the YPDP SharePoint site maintained by the evaluation contractor is available to each YPDP site. This training module provides an overview and basic instructions for entering data into the PTS for both new users and as a refresher for existing system users.
Telephone Follow-up Surveys. The data collection of the telephone follow-up surveys will be accomplished through the use of CATI. CATI systems collect responses 100 percent electronically. The systems also perform a number of functions to avoid errors including:
providing correct question sequence;
automatically executing skip patterns based on prior question answers (which decreases overall interview time and consequently the burden on respondents);
recalling answers to prior questions and displaying the information in the text of later questions;
providing random rotation of specified questions or response categories (to avoid bias);
ensuring that questions can not be skipped; and
rejecting invalid responses or data entries.
The CATI system lists questions and corresponding response categories automatically on the screen, eliminating the need for interviewers to track skip patterns and flip pages. Moreover, the interviewers enter responses directly from their keyboards, and the information is automatically recorded in the computer’s memory. CATI allows the computer to perform a number of critical quality assurance routines that are monitored by survey supervisors, including tracking average interview length, refusal rate, and termination rate by interviewer, and performing consistency checks for inappropriate combinations of answers.
4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in item 2 above.
The information being recorded into the PTS for the YPDP evaluation is not otherwise available in the format required for conducting secure and accurate random assignment in a systematic manner across sites. Each grantee submits quarterly reports to ETA; however, the summary program reports present financial information only and exclude participant details needed to conduct an impact analysis and program descriptive information needed to conduct a process/implementation analysis. With the exception of earnings data available from the Unemployment Insurance wage record system maintained by states (which will be an additional source of data that will be used in the YPDP evaluation effort), there are no other sources of information that could be used in place of the PTS and the follow-up surveys at 18 and 36 months to systematically assess outcomes for YPDP treatment and control group members. The administrative data available do not permit systematic and timely analyses of differences between treatment and control groups on, for example, hourly wages, number of hours worked each week, views on helpfulness of services, changes in housing status, changes in family composition, views on quality of mentoring services received, and other topical areas that are the focus of the follow-up survey. In addition, the 18-month follow-up survey will provide a baseline of data that can be compared to those of the 36-Month follow-up survey to determine the extent to which YPDP net impacts (if they are in evidence at 18 months) persist or erode over time. The existing data are not satisfactory to report on the impacts of YPDP services and the extent to which such services make a statistically significant difference in terms of employment, earnings, educational achievement, and parenting outcomes.
5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize burden.
The PTS, YPDP implementation field-based data collection, and 18- and 36-Month follow-up surveys do not impact small businesses or other small entities (other than those that are YPDP grantees). The impact on grantee organizations is minimal, involving staff input of participant information into the web-based PTS -- about 1 hour staff time per month -- and meeting with Urban Institute evaluators during the site visits (about 45 minutes of time for each of up to 8 staff members and/or administrators.).
6. Describe the consequences to federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles in reducing burden.
Participant Tracking System (PTS). The consequences of not collecting information through the PTS are: (1) the integrity of the random assignment process would be compromised if grantees were to handle this activity directly; and (2) valid and comparable data would not be available for analysis across the YPDP sites relating to numbers and types of individuals served, services received, and outcomes.
Implementation Site Visits. The consequences of not conducting the field-based implementation and process analysis is that there will be a lack of in-depth information about the nature of the strategies developed and employed at YPDP sites to improve the employment, education, and training outcomes of young parents. Site visits will provide an opportunity to fully document the services being delivered to treatment and control group members, which is an essential part in an experimental design for understanding, for example, if employment outcomes at various points after random assignment are potentially associated with varying services received by treatment and control group members. If there are positive net impacts for the treatment group, it will be vital to understand the specific intervention(s) received by treatment group members so that they could potentially be replicated by other employment and training programs serving parenting youth.
Follow-up Surveys. Without the follow-up surveys, ETA would not have the full range of outcome information needed to gauge the net impacts of YPDP services on treatment group members. For example, the follow-up surveys conducted 18 and 36 months after random assignment will allow for systematic comparison of employment, earnings, education, and parenting outcomes for the treatment and control group members to determine net impacts of the YPDP. The surveys will also provide valid and reliable data for understanding variation in types of services received by YPDP participants and the views of YPDP participants concerning the usefulness of services received. These surveys will also allow for comparison over time of key outcomes to determine if impacts detected at 18 months after random assignment are sustained at 36 months after random assignment.
Overall, the federal investment of resources into the YPDP and the requirement to conduct a rigorous evaluation of the implementation strategies and their impacts requires the systematic collection of program and participant data. The integrity of the random assignment process requires a secure and technically sound tracking system. With significant expenditures involved in implementing the YPDP, it is critical to document the different models and projects that are operating under the initiative, examine and assess the implementation to date, and identify innovative features and potentially promising strategies. The site visits are critical to this evaluation project, as it represents the only opportunity to gather comprehensive information on implementation from all grantees. The survey is an essential component of the evaluation project because it enables the evaluation team to systematically assess and identify which of the strategies have been useful for the participants and to associate service participation with the outcome measures of interest. Given the significant expenditures involved in implementing employment, training, and other human service programs for parenting youth, as well as the importance that Congress, DOL/ETA, and the public places on cost-effectively serving parenting youth, the proposed data collecting effort is of critical importance.
7. Explain any special circumstances that would cause an information collection to be conducted in a manner:
requiring respondents to report information to the agency more often than quarterly;
requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
requiring respondents to submit more than an original and two copies of any document;
requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years ;
in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
requiring the use of statistical data classification that has not been reviewed and approved by OMB;
that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
There are no special circumstances that would cause this information collection to be conducted in any manner listed above.
8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.
Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.
Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years – even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.
Notification of this survey was published in the Federal Register, Vol. 76, No. 50, (Tuesday, March 15, 2011: pp. 14099 – 14100), a copy of which is in Attachment H. Readers were given 60 days from the date of publication to submit comments. No comments were received.
9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
Grantee administrators and staff using the PTS or being interviewed during site visits will receive no payments or gifts. Respondents to the follow-up surveys will receive $15 upon completion of the survey. Research suggests that such incentive payments can make a critical difference in boosting response rates in telephone surveys. The likelihood of survey cooperation increases when the respondent perceives a benefit from the survey and the cost of completing the survey is not burdensome. Research suggests that incentives may persuade sample members who have otherwise little reason to participate in the survey. Providing a small incentive adds to the perceived benefits of completing the survey and adds to the legitimacy of the request. A “token” monetary incentive works best because larger amounts are likely to be confused with a payment. (Singer, 2002, Dillman, 1978) Although we anticipate that most participants in the YPD program will have at least some interest in the survey topic due to their participation in the
demonstration program, we also anticipate that some will be reluctant or skeptical or have time constraints. We suggest $15 as sufficient to entice reluctant respondents, to add legitimacy to the request for those that may have time constraints and to compensate for respondent’s time and effort expended in responding to the survey.3
10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
No assurances of confidentiality are provided for data entered into the PTS or to respondents interviewed during site visits. However, survey respondents will be informed that the highest standards will be applied to protect the privacy of all data collected.
Participant Tracking System (PTS). Information on individual participants entered into the PTS will be subject to the highest standards of privacy. (See Attachment A for details of the system and screen shots of the PTS, including the specific data elements being collected through the system on each YPDP participant). The web-based application will reside on a standard Windows server running a web service such as IIS or Apache that will be physically located in a monitored, access-controlled, secure server room at The Urban Institute in Washington DC. The web server has been hardened using a best practice security hardening checklist. The web server will be dedicated solely to the PTS application. Administrator access to the database server will be restricted to an authorized Urban Institute server administrator. Accounts on the web server will be password protected. Passwords will be at least 8 characters long and contain at least 1 special character and number and will not contain dictionary words. These requirements are enforced upon account creation. Passwords will expire every 90 days and users will have to create new passwords that fulfill the requirement of the password policy. A dedicated Secure Site License (SSL) will be employed for this project and all non-encrypted access to the MIS application will be restricted. Any logging or output files will not contain personal identifications (PI) and will be limited to generic system and error data. All data transmitted from the grantee sites to the application server will take place over the encrypted SSL connection. Data extracts for use by the project team will be available in files encrypted with the PGP software and available to project team members on the Urban Institute’s SecureFTP (SFTP) site.
The system will segregate user data into PI data and project specific data. PI data such as social security numbers will be stored in a separate database table containing a system generated ID number with the social security number stored in encrypted form. PI data will be entered into the system but will at no point be displayed or downloadable by users of the system. PI data will be stored separately from project specific data and will be available for updating only by the grantee that originally entered the data. Project specific data will be available to the project team in specific extracts and reports.
Implementation Site Visits. The administrators and staff interviewed by evaluators when on-site will be assured that their responses will not be identified by the individual in any reports nor will interview notes be shared with ETA Individuals will be interviewed separately and in private offices. (See protocol in Attachment B for the statement that will be used during site visits to assure respondents of privacy.) To preserve privacy, paper copies of interview notes will be secured in a locked file cabinet. If any notes are recorded on laptop computers, such notes will be stored in a SQL Server database located in an access-controlled server room at the Urban Institute.
Telephone Follow-up Surveys. The YPDP participants surveyed will be assured that their responses will be kept private within the limits of the law (note: see survey introduction in Attachments C and D which provides client assurance). The survey data will be stored on an evaluation contractor computer that is protected by a firewall that monitors and evaluates all attempted connections from the Internet. PI data on each survey respondent will be maintained in a separate data file and apart from the survey data so that it is not possible to link particular responses to individual respondents. Once the survey is completed, all PI data on each respondent will be destroyed. The entire survey database will be encrypted so that any data stored will be further protected. Finally, access to any data with identifying information will be limited only to contractor staff directly working on the survey. All findings in any written materials or briefings will be presented at the aggregate level and it will not be possible to link specific responses to individual respondents in any way. The contractor will not include any identifying information such as names, addresses, telephone numbers, or social security numbers in the database delivered to DOL. The basis for the assurance of confidentiality for the 18- and 36-month follow-up surveys is the Privacy Act.
11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers these questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
Participant Tracking System (PTS). The PTS, as detailed in Attachment A, records individual identifying information of a sensitive, personal, and private nature, including (1) last and first name; (2) Social Security number; (3) date of birth, (4) ethnicity and race; (5) marital status; (6) whether the individual is pregnant or expecting a child; (7) number of children; (8) whether the individual is a TANF or SNAP recipient; (9) name and contact information for three individuals who can be contacted if the program cannot locate the individual; and (10) hourly wage at 6, 12, and 18 months after random assignment.
The reasons why these questions are considered necessary and specific uses to be made of this information:
The last and first names are needed by YPD site staff to be able to effectively use and update the data system in a reliable, efficient, and user-friendly manner (e.g., to be able to easily locate an individual’s record and update the record with service and outcome data).
The Social Security Number (SSN) of each YPD participant is needed by YPD grantee sites for performance management purposes. The SSN data are immediately encrypted upon entry into the PTS. SSN is included in the PTS to enable sites to use the PTS as a single, stand-alone automated case management system. In addition, the collection of SSN will allow the possibility (at a later date) for individual grantee sites to match individual participants with other administrative data (such as data they may maintain in other data systems managed by the grantee or with other external administrative records systems, such as unemployment insurance wage record data collected by state workforce agencies).
Other data which are of a sensitive nature – including date of birth, ethnicity, race, marital status, whether the individual is pregnant or expecting a child, number of children, and whether the individual is a TANF or SNAP recipient -- are needed to support detailed analyses of the types of youth in the treatment and control groups receiving YPD services, as well as to conduct analyses of services received and educational/employment outcomes by various subgroups. In addition, YPD sites require data such as, the date of birth, number of children, and whether the individual is pregnant or expecting a child, to determine whether individuals are eligible to participate in YPD (e.g., are between 16 and 24 years of age and are parenting or expecting a child).
Name and contact information for up to three individuals who know the participant are necessary in order for the YPD site administrators and staff to locate the participant should they lose contact to help ensure the program is able to continue to provide YPD services, to reduce attrition from the program, and to help sites locate participants to collect outcome and exit information (e.g., employment status at 6, 12, and 18 months after random assignment).
Hourly wages for each YPD participant at 6, 12, and 18 months after random assignment are needed as an important short-term employment outcome measure.
Prior to participating in YPD, demonstration site staff provide a description of the YPD program initiative, and the informed consent and random assignment process. All applicants must read and sign an informed consent form prior to being randomly assigned. If the individual is younger than 18 years of age, her/his parent must also sign this form. A copy of the informed consent form for YPD participants is attached at the end of Attachment A, along with a sample of the PTS forms being used. All participants are also informed that collection of data for the PTS is voluntary and they are not required to answer any of the questions on the PTS forms.
Implementation Site Visits. There are no data of a sensitive, personal, or private nature collected in the site visit interview guide.
Telephone Follow-up Surveys. There are several questions that are of a sensitive, personal, and private nature that will be collected as part of the 18- and 36-month follow-up surveys. The following questions may be considered as sensitive, personal or private by participants (see Attachments C and D for the specific questions): (1) regular hourly pay for the participant’s current or most recent job (Question 32 and 33); (2) receipt of cash assistance from a state or county welfare program (Question 37), type of assistance received (Question 38), and months of cash assistance received over the past 18 months (Question 39); receipt of SNAP or food stamp benefits (Question 40); receipt of Medicaid or similar state health program (Question 44) and whether children were covered by Medicaid or similar state health program (Question 45); number of people living with the participants and changes in family composition over the past 18 months (Questions 48 through 52); marital status and whether it has changed over the past 18 months (Questions 53 and 54); whether the individual is currently living with her/his spouse or partner (Question 55) or her/his parents (Question 56); how many biological children the participant has living in her/his household (Question 57) and whether and how many children live elsewhere (Questions 58 and 59); whether the participant receives or pays child support, and if so, number of payments and amounts received over the past 18 months (Questions 60 through 65); extent of the participant’s relationship and engagement with her/his children (Questions 66 through 69); whether the participant was unable to pay her/his mortgage, rent or utility bills during the past 18 months (Question 73) and whether the participant or her/his children moved in with other people during the past 18 months because of inability to pay mortgage, rent, or utility bills (Question 74); and the combined income of all members of the participant’s family during the past 18 months (Question 77).
Each of the data items identified above to be collected at 18 and 36 months after random assignment are necessary to evaluate the differences in critical outcomes for parenting youth enrolled in the treatment and control groups. The data will permit analyses of impacts between the control and treatment groups on a range of outcome variables that would not otherwise be available or possible without the follow-up survey data, including changes in hourly wages, earnings, and family income; receipt of public assistance; marital status and family composition; and the ability to meet food and housing needs of the family. Combined with data collected as part of the PTS, it will also be possible to conduct subgroup analysis of participant outcomes (e.g., for different participant characteristics at the time of random assignment). As part of the informed consent process prior to random assignment participants are asked to provide consent for collection of follow-up data and informed about the likelihood that one or more follow-up surveys will be conducted. They are also informed at the time of random assignment that such follow-up surveys are voluntary on their part and that they may or may not provide answers to some or all of the survey questions. Additionally, as reflected in the introduction to the follow-up survey, participants are again informed that the survey is voluntary on their part and that their responses to each question will remain confidential within the limits of the law (see introduction to each of the follow-up surveys in Attachments C and D for the explanation of the purposes of the survey, its voluntary nature, and assurances of confidentiality to be provided to each survey respondent).
12. Provide estimates of the hour burden of the collection of information. The statement should:
Table 2 provides the estimate of the respondent hour burden for (1) collecting data from treatment and control group participants for entry into the Participant Tracking System; (2) conducting interviews with YPDP site administrators and staff during the process/implementation site visits to the four YPDP sites, and (3) conducting the follow-up surveys with YPDP treatment and control group participants in mentoring sites at 18 and 36 months after random assignment into YPDP. Across these data collection efforts, the total hour response burden is 2,527 hours.
The estimate for the hour burden for collection of participant data for the Participant Tracking System (PTS) is 1,633 hours. Enrollment in YPDP is expected to be 1,633 treatment and control group participants. Data will be collected and entered into the PTS on all YPDP participants at six points in time averaging approximately 10 minutes each: (1) at intake, (2) at random assignment, (3) to record service receipt, and (4) to record employment outcomes at 6, 12, and 18 months after random assignment. Total hours burden per YPDP participant is estimated at one hour. With PTS data being collected on all YPDP participants (100 percent response rate), the total estimated response burden for the PTS data collection is 1,633 hours.
TABLE 2: ESTIMATED RESPONSE BURDEN IN HOURS OF DATA COLLECTION
Data Collection Activity |
Number of Respondents |
Frequency |
Time Per Respondent |
Total Burden Hours |
PTS – Data Collection |
1,633 |
Six |
60 minutes |
1,633 |
Site Visit Interviews |
32 |
Once |
45 minutes |
24 |
18-Month Survey |
1,306 |
Once |
20 minutes |
435 |
36-Month Survey |
1,306 |
Once |
20 minutes |
435 |
Total |
4,277 |
|
35 minutes |
2,527 |
The Implementation Field Site Visit Discussion Guide will be used to interview an average of eight administrators and staff in each of the 4 YPDP sites – a total of 32 respondents. The estimated response rate is 100 percent, since when arranging for the site visits, evaluators will confirm scheduled times for interviewing key administrators and staff. The estimated response time for the interviews is an average of 45 minutes. Total estimated response burden for the site visits is 24 hours.
The 18- and 36-month follow-up surveys will each be conducted with a total of 1,633 YPDP participants at the YPDP sites with mentoring bump-up services. The estimated response rate is 80 percent – yielding valid responses from a total of 1,306 YPDP participants to each survey. The estimated response time for the survey is 20 minutes. This will result in a total estimated response burden for each survey of 435 hours.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage and rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 13.
The total annualized cost to respondents for the survey is presented below in Table 3. The total estimated costs for these data collection activities are $47,490. The average hourly wage in that table for the PTS and follow-up surveys is $18.76, based on the BLS average hourly earnings of production and nonsupervisory employees on private, service providing, nonfarm payrolls (September 2010 National Industry-Specific Occupational Employment and Wage Estimates, from the U.S. Department of Labor, Bureau of Labor Statistics and available on the department’s website).4 The hourly wage in that table for the interviews with YPDP administrators and staff during the YPDP site visit is the average hourly wage of $22.21, based on the BLS average hourly earnings of all employees on private, service providing, nonfarm payrolls (September 2010 National Industry-Specific Occupational Employment and Wage estimates, from the U.S. Department of Labor, Bureau of Labor Statistics and available on the department’s website.)5
TABLE 3: TOTAL ANNUALIZED COSTS OF DATA COLLECTION FOR RESPONDENTS
Data Collection Activity |
Total Burden Hours |
Average Hourly Wage |
Total Annualized Cost |
PTS – Data Collection |
1,633 |
$18.76 |
$30,635 |
Site Visit Interviews |
24 |
$22.21 |
$533 |
18-Month Survey |
435 |
$18.76 |
$8,161 |
36-Month Survey |
435 |
$18.76 |
$8,161 |
Total |
2,527 |
|
$47,490 |
13. Provide an estimate for the total annual cost burden to respondents or record-keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).
The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services component. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life of capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling and testing equipment; and record storage facilities.
The proposed data collection for the PTS, YPDP on-site visits, and follow-up surveys will not require respondents to purchase equipment or services or to establish new data retrieval mechanisms. There are no capital/start-up or ongoing operation/maintenance costs associated with this information collection. The field-based implementation/process data collection involves semi-structured interviews discussing staff and administrators’ descriptions of services and service delivery, and their experiences, opinions, and factual information. The follow-up survey content is based on the respondents’ experiences, opinions, and factual information. Therefore, the cost to respondents solely involves the time involved in being interviewed or in entering data into the PTS. These costs are captured in the burden estimates provided in A.12.
(a) We do not expect any total capital and start-up costs.
(b) We do not expect extensive time spent on generating, maintaining, and disclosing or providing the information.
14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.
TABLE 4: TOTAL ANNUALIZED COSTS TO THE FEDERAL GOVERNMENT
Year |
Dates |
Costs |
1 |
2011-2012 |
$1,114,000 |
2 |
2012-2013 |
$1,114,000 |
3 |
2013-2014 |
$1,114,000 |
4 |
2014-2015 |
$1,114,000 |
5 |
2015-2016 |
$1,114,000 |
Total |
$5,570,000 |
The complete cost of the three data collection activities, which are the subject of this submission, is estimated at $4,456,000, including contract staff salaries. The costs of the three data collection activities (i.e., development/implementation of the PTS, conduct of implementation site visits, and conduct of the 18- and 36-month follow-up surveys) are estimated at 80 percent of the total evaluation budget. Thus, if $4,456,000 divided by five years is annualized at $891,200.
15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I.
This is a new request.
16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and end dates of the collection of information, completion of report, publication dates, and other actions.
The analysis that will be involved in the YPDP evaluation will focus on the qualitative field-based implementation analysis, descriptive summary tabulations of participant data drawn from the PTS, and estimation of net impacts of YPDP services based on data collected from the follow-up survey and Unemployment Insurance wage record data. Together, the quantitative and qualitative analysis will describe the structure, operations, and service delivery in the YPDP grant-funded programs; document the pre-existing services and the mentoring interventions; present tables showing the demographic characteristics of the treatment and control group members; and estimate net impacts of YPDP services on participant employment, earnings, education, and parenting outcomes. Because the YPDP evaluation employs an experimental design with YPDP participants being randomly assigned to treatment and control groups, the analysis of net impacts will be statistically straightforward (e.g., using t-tests to determine if the means for the treatment and control groups are statistically different on, for example, percent employed or total earnings at 18 months after random assignment). Data will be presented in summary formats such as tables, charts, graphs that allow ETA and other stakeholders to easily understand the range of YPDP programs, the bump-up models employed and their implementation, number and characteristics of YPDP participants, service utilization, participant outcomes, and net impacts.
The time schedule for the project, including beginning and end dates of the collection of information and completion of reports are as follows (note: reports are asterisked):
YPD Program or Evaluation Activity |
Completion Date |
Grants Issued |
6/30/2011 |
Enrollment Begins |
10/1/2011 |
Site Visits Begin |
10/1/2012 |
Site Visits End |
3/31/2013 |
Enrollment Ends (2 years) |
9/30/2013 |
18-Month Follow-up Survey Completed |
9/30/2014 |
Interim Report Submitted (based on PTS, Site Visits, 18-Month Survey,) |
12/31/2014 |
36-Month Follow-up Survey Completed |
3/31/2016 |
Final Report Submitted (PTS, Site Visits, 18- and 36-Month Surveys) |
6/30/2016 |
Preliminary analysis of data collected through the PTS, site visits, and surveys will begin shortly after each of these data collection efforts are initiated. For example, as data are entered into the PTS it will be possible to conduct frequency distributions and cross-tabulations to (preliminarily) examine participant characteristics, service utilization, and interim outcomes. Similarly, in using the CATI system to conduct the follow-up survey, it will be possible to tally results on each survey question as the survey sample builds. Once all of the YPDP participants have been enrolled, fully served, and exited from YPDP services and follow-up data on employment has been entered into the system, the evaluation team will be able to conduct complete analyses of the PTS.
One interim report and a final evaluation report are planned. The interim report will present evaluation results for grantees, providing results based on the site visits, the PTS and the 18-month follow-up survey and will be submitted by December 31, 2014. A final evaluation report, providing results based on the site visits, the PTS, and 18-month and 36-month follow-up surveys will be submitted by June 30, 2016. The interim and final reports will categorize grant strategies and bump-up models; analyze enrollment and participant characteristics; assess implementation success and challenges; analyze service utilization/receipt; provide overall, and by YPDP site, in-depth analyses of participant outcomes for all YPDP sites; estimate net impacts at mentoring sites only; and provide study conclusions and recommendations. The interim and final reports will be submitted to ETA in Microsoft Word and PDF format (for publication on the ETA website). In addition, public use data files, stripped of individual identifiers, will be submitted to ETA with the final report which will include (1) the participant data entered into the PTS and (2) the 18-month and 36-month follow-up surveys.
17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
The OMB approval number and expiration date will be displayed or cited on all information collection instruments.
18. Explain each exception to the certification statement identified in Item 19,
“Certification
for Paperwork Reduction Act Submissions,” of OMB Form 83-I.
There are no exceptions to the certification statement.
1 U.S. Department of Labor, Employment and Training Administration. 2008. “Young Parents Demonstration Program (YPDP) SGA/DFA PY 08-08,” Federal Register, Vol. 73, No. 193, October 3, 2008 (available over the Internet at: http://edocket.access.gpo.gov/2008/pdf/E8-23319.pdf).
2 Workforce Investment Act of 1998, Title I, Subtitle D, Section 171 (b), Public Law 105-220. See Attachment G for the authorizing language and Sections 171 and 172 that authorize evaluations, including those of demonstrations.
3Dillman, D. A. (1978), Mail and Telephone Surveys: The Total Design Method, NY: John Wiley and Sons. Inc. Singer, Eleanor (2002), "The Use of Incentives to Reduce Nonresponse in Household Surveys," in Groves, et al. Survey Nonresponse (New York: Wiley and Sons, 2002). Additional studies assessing the utility of incentive payments for boosting survey response rates include: Singer, Eleanor, John Van Hoewyk, Nancy Gebler, Trivellore Raghunathan, and Katherine McGonagle (1999), “The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys,” Journal of Official Statistics 15: 217-230; and Singer, Eleanor, John Van Hoewyk, and Mary P. Maher (2000), “Experiments with Incentives in Telephone Surveys,” Public Opinion Quarterly 64:171-188.
4 U.S. Department of Labor, Bureau of Labor Statistics, Table B-8. Average hourly and weekly earnings of production and nonsupervisory employees on private nonfarm payrolls by industry sector, seasonally adjusted (accessed from the following website as of September 2010: http://www.bls.gov/webapps/legacy/cesbtab8.htm)
5 U.S. Department of Labor, Bureau of Labor Statistics, Table B-3. Average hourly and weekly earnings of all employees on private nonfarm payrolls by industry sector, seasonally adjusted (accessed from the following website as of September 2010: http://www.bls.gov/webapps/legacy/cesbtab3.htm)
A-
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 1995 |
| Author | Administrator |
| Last Modified By | Naradzay.Bonnie |
| File Modified | 2012-05-04 |
| File Created | 2012-05-04 |