national evaluation of P3 omb supporting statement: part b
The Chief Evaluation Office (CEO) of the U.S. Department of Labor (DOL) has contracted with Mathematica Policy Research and its partner Social Policy Research Associates (hereafter “the study team”) to conduct the National Evaluation of the Performance Partnership Pilots for Disconnected Youth (P3). The evaluation will provide information to policymakers and administrators about how the nine Cohort 1 pilots and six Cohort 2/3 pilots used the flexibility offered through P3 to pool funds and waive programmatic requirements and how this flexibility helped them overcome significant hurdles in providing effective services to and improving outcomes for disconnected youth. In addition, the evaluation represents an important opportunity to study the implementation outcomes and system changes that the pilots are able to achieve and the outcomes of and impacts on youth participants of the pilot programs.
This package requests clearance for four data collection activities conducted as part of the evaluation’s implementation and systems analysis: (1) site visit and phone interviews; (2) focus group discussions with P3 youth participants; (3) a survey of partner managers; and (4) a survey of partner service providers or partner network survey.
In October 2015, nine competitively awarded grantees were announced as the first P3 cohort. They received up to $700,000 in start-up funds and the flexibility to blend or braid discretionary funds from fiscal year 2014 and 2015 to improve the outcomes of disconnected youth. The first cohort grantees are located in eight states (California, Florida, Illinois, Indiana, Kentucky, Louisiana, Oklahoma, and Washington) and a federally recognized Indian tribe located in Texas. Subsequently, six Cohort 2/3 grantees representing four states (Arizona, California, Connecticut, and New York) were awarded P3 grants. As required in the legislation authorizing P3, the grantees are serving disconnected youth, defined as low-income youth ages 14 to 24 and are either homeless, in foster care, involved in the juvenile justice system, unemployed, or not enrolled in or at risk of dropping out of school. Several grantees are serving in-school and out-of-school youth, and some are focusing on specific populations such as youth in foster care or public housing. Almost all of the Cohort 1 grantees are relying on their Workforce Innovation and Opportunity Act Title I Youth funds along with other DOL, Department of Education (ED), Department of Health and Human Services (HHS), Corporation for National and Community Service (CNCS), and Institute of Museum and Library Services (IMLS) funds. Table B.1 provides additional information about the 15 pilots. Table B.2 provides sample sizes for the four data collection efforts.
Table B.1: Description of the 15 pilots
Pilot name |
Location of pilot services |
Anticipated number of participants |
Estimated number of partners |
Target population |
Brief description of intervention |
|||||
Cohort 1 |
||||||||||
Baton Rouge P3 |
Baton Rouge, Louisiana |
84 |
6 |
14 to 24 year-olds who are 2 or more years behind in school |
Youth will develop an individual success plan. Program staff will develop training activities, and will encourage youth to participate in other training and education programs provided by partners. |
|||||
Best Opportunities to Shine and Succeed (BOSS) |
Broward County, Florida |
420 |
4 |
At-risk youth in six high schools |
Students will be provided a case manager. The case managers will have a 1:35 ratio of case manager to youth. The case manager will connect each participant with “evidence-based and evidence-informed” educational, employment, and personal development services that specifically address the needs of the student in regard to graduation and post-secondary success. The BOSS program will provide intensive, comprehensive, and sustained service pathways via a coordinated approach that helps youth progress seamlessly from high school to post-secondary opportunities. |
|||||
Chicago Young Parents Program (CYPP) |
Chicago, Illinois |
140 |
3 |
Low-income women ages 16 to 24 with at least one child younger than six |
CYPP is a parent engagement, education and employment program that combines two successful, research-based program models: employment and mentoring for youth and high quality comprehensive Head Start programming for children and families. All participants receive basic Head Start services plus additional mentoring, home visits, socializations, education planning, enrichment sessions, and employment. |
|||||
Indy P3 |
Indianapolis, Indiana |
80 |
8 |
At-risk, low-income youth ages 14-24; target youth in public housing |
Indy P3 will provide comprehensive, concentrated, and coordinated services for cohorts of very high risk disconnected youth. Staff members called connectors (each serving 40 youth and families at a time) will develop individual service and success plans, link participants to core service providers, and share data across programs. Partners will emulate best practices and lessons learned from evidence-based models. |
|||||
Los Angeles P3 (LAP3) |
Los Angeles, California |
8,000 |
24 |
Youth ages 16 to 24 |
LA P3 is comprehensive service delivery system that coordinates and integrates the delivery of education, workforce, and social services to disconnected youth. Partner agencies and WIOA youth contractors in the city of Los Angeles provide the program services. These are existing services: the aim of LAP3 is to enhance the availability of these services through the enhanced coordination of partner agencies. |
|||||
P3-OKC |
Oklahoma County, Oklahoma |
60-70 |
12
|
Foster youth ages 14 to 21 |
Youth will receive: (1) modified wraparound services more consistent with child welfare services; (2) an integrated plan of services to promote service integration and foster partnerships across nonprofit and public organizations; (3) the Check and Connect intervention designed to monitor school attendance, participation, and performance; and (4) enhanced vocational development, work, and/or career opportunities achieved through wraparound, educational options, and career aspects of students enrolled in career academies. |
|||||
Seattle-King County Partnership to Reconnect |
Seattle-King County, Washington |
200 |
3
|
Youth ages 16 to 24 |
The program will have three components: (1) strategic coordination of workforce development services with the state’s unique Open Doors policy, which provides K-12 funding for reengagement programs; (2) utilizing AmeriCorps members to develop a regional outreach strategy aimed at placing the hardest to serve youth in programs that best reflect their interests and needs; and (3) advancing efforts toward a shared data system and common intake process that will enhance the coordination and targeting of services across Seattle-King County. |
|||||
Southeast Kentucky Promise Zone P3 |
7 rural southeast Kentucky counties |
1,000 |
3 |
At-risk youth ages 14 to 24 |
The program will include a teen pregnancy prevention program, career assessments and exploration trips, academic and career mentoring and tutoring, professional development for teachers and community members, two generations of family engagement focused on youths’ parents, and paid work experience. |
|||||
Tigua Institute of Academic and Career Development Excellence |
Ysleta Del Sur Pueblo tribe (Texas) |
45-50 |
2 |
Tribal youth members ages 14 to 17 enrolled in two local high schools |
Youth will receive group sessions of an integrated Leadership curriculum based on nation building theory and the Pueblo Revolt Timeline, which includes the Tigua lecture series to teach youth about their history, language and tribal government and the various services offered by the departments. Youth will also receive individually based wraparound services. |
|||||
Cohort 2/3 |
||||||||||
Partnerships Advancing Youth Together in Hartford |
Hartford, Connecticut |
100 |
5 |
Disconnected youth ages 16 to 24 residing in North Hartford Promise Zone |
Promise Zone PATH tests an innovative strategy of reconnecting youth to success by creating a virtual network between programs and connecting databases and applications of existing evidence-based programs. The dashboard identifies disconnected/opportunity youth experiencing gaps in service, match them to needed services and programs, and track their progress on a pathway to success that is sustainable at the youth, program, provider and system levels. |
|||||
LA County P3 |
Los Angeles County, California |
963 |
4 |
18 to 24 year olds |
Public Service Career Mentoring for System Involved Youth has two levels. Level 1 focuses on preparing young adults for employment. It consists of curriculum and paid work experience that will be completed in about 6 months. Young adults who complete Level 1 may apply for Level 2, which is a competitive one-year internship with a county office. The young adult is enrolled in WIOA and has a career mentor. |
|||||
New York City P3 |
Brooklyn, NY |
50 |
3 |
Young parents ages 17 to 24 and their children |
The program seeks to improve participants’ access to multiple career pathways through activities including guaranteed childcare, high school equivalency preparation, work readiness and occupational skills training (including paid work experiences), supportive services, and parenting support. The program provide personalized enrichment and support plans for participants through a Child Care Navigator. |
|||||
New York State |
Albany, New York |
120 |
2 |
14 to 24 year olds who are unemployed, not in school, or reentering the community from juvenile justice services. |
The Connecting Youth in Transition intervention uses integrated case management, working with transition coordinators to jointly develop individual service plans. Youth will be linked with existing educational and employment services, and the coordinator follow up and track progress for up to four months. |
|||||
Phoenix Manufacturing Apprenticeship Program |
Phoenix, Arizona |
96 |
6 |
Young adults ages 17 to 24 disconnected from school and work settings |
The program is a workforce development program where participants will receive two curricula (the Gateway College Manufacturing Curriculum and the TCI Solutions soft skills curriculum). |
|||||
Sacramento Promise Zone Performance Partnership Pilot |
Sacramento, California |
100 |
9 |
Youth ages 16 to 24, who are homeless, aging out of foster care and/or on probation |
The pilot provides young adults with a housing voucher, continued case management from the referring agency, and additional services to help them accomplish their education and employment goals. They also receive mental health services to support their transition to a new living situation. |
|||||
Sources: Cohort 1: Grantee presentations at Annual P3 Conference in June 2016; grantee draft evaluation plans; and grant applications. Cohort 2/3: grant application and telephone discussions.
Table B.2. Sample sizes by data collection activity
Respondents |
Total number of respondents over evaluation |
Number of responses per respondent |
Site visit and phone interviews |
|
|
Administrators and staff (Cohort 1 site visits) |
126 |
2 |
P3 Program directors (Cohort 1) |
9 |
3 |
Administrators and staff (Cohort 2/3) |
90 |
1 |
Administrators and staff (Exemplar) |
90 |
1 |
Focus group discussions |
|
|
Youth (Cohort 1) |
216 |
1 |
Youth (Cohort 2/3) |
48 |
1 |
Survey of partner managers |
|
|
Partner managers (Cohort 1) |
90 |
2 |
Partner managers (Cohort 2/3) |
60 |
1 |
Partner network survey |
|
1 |
Frontline staff (Cohort 1) |
90 |
1 |
Front line staff (Cohort 2/3) |
60 |
1 |
Total |
879 |
-- |
All 15 pilots of Cohorts 1 and 2/3 are included in the evaluation. In addition to the site visits that occurred to the nine Cohort 1 pilots in 2017, we will conduct second round visits to Cohort 1 grantees in Spring 2018 and final telephone interviews with program directors in Fall 2018. In addition, we will conduct site visits to the six Cohort 2/3 pilots in Fall 2018, as well as visits to six exemplary grantees in late 2018.During the site visits, we will interview grant and program administrators and managers of partner organizations as well as frontline staff of P3 partner agencies that are providing P3 services to youth. To the extent we can, we will interview all appropriate staff but, given constraints on the length of visits and respondents’ other responsibilities, we might need to select one or two representatives from a particular staff position from among the larger group of staff who would be suitable to interview.
The study team will conduct an average of three focus groups of participants at each of the nine Cohort 1 pilots across the two site visits conducted to those pilots. The study team will conduct a single focus group during each of the six Cohort 2/3 visits. We will not conduct focus groups during exemplar visits. In Cohort 1pilots providing different types of services to distinct groups of participants—for example, in-school and out-of-school youth—we will conduct separate focus groups.
Focus group participants will be a convenience sample of youth who are receiving services and are willing to participate. We will ask the pilot staff to recruit P3 program participants for the focus groups. We will request that, to the extent possible, the participating youth be diverse on characteristics such as gender, age, race/ethnicity, and length of program participation to provide different perspectives on the program.
Participants of these focus groups are not intended to be representative of experiences of all youth within a pilot and are subject to bias given staff recruitment methods and youths’ own decision whether to participate. Conducting focus group with the convenience sample is to provide additional perspectives on the P3 program that is unique from what we will collect from program staff or partners. The data collection is intended to capture a potential range of experiences within the pilots to identify themes in successes, challenges, and possible improvements, rather than a sample that is statistically representative of a larger group. Prior experience has indicated that, for providing additional insights into a research question, a convenience sample is sufficient.
During all site visits except exemplar visits, we will administer the partner manager survey to all managers or leaders of the P3 pilot partner agencies and organizations. We will request that they complete the short survey after concluding the on-site interview. The survey will provide quantifiable information about respondents’ perceptions of the P3 collaboration.
The survey is a brief, targeted tool to explore the strength of relationships between the key entities (partners) that oversee service delivery within the local P3 system. We expect to administer the survey to the main staff person of all identified service delivery partners of each pilot, up to a maximum of 15 per pilot. During the 2018 Cohorts 1 and 2/3 visits, prior to survey administration, we will work with each partner to identify the appropriate survey respondent.
The data sources and data collection activities for the National Evaluation of P3 are as follows.
Site visits and interviews to pilots. We visited each Cohort 1 pilot in 2017; we will conduct a second visit in 2018. Additionally, in Fall 2018 we plan to conduct interviews with Cohort 1 program directors. We will visit Cohort 2/3 pilots once in 2018. We will conduct visits to six exemplar pilots in 2019. Before each pilot’s first site visit, the study team will collect and assess the key information previously collected about the pilot, such as the pilot’s partners and the services provided to its youth participants. Part A of this submission lists the research topics that the study team will explore during site visits; the main objectives are to collect and analyze data on (1) the local systems created for P3 and (2) how P3 flexibility has resulted in different and innovative services to disconnected youth.
To capture multiple perspectives, the study team will obtain information on each topic from multiple respondents so that no single person’s opinions or responses will be assumed to be fully representative. The study team will interview both managers and frontline staff in an effort to understand not only how service delivery and administrative processes are supposed to work, but also how they actually work. These respondents will include grantee lead(s), pilot manager(s), data systems manager, partner managers, and frontline staff providing services.
The master site visit and interview protocol (see Instrument 1) will guide on-site and telephone interviews. The specific questions and length of each interview will depend on the specific respondent. Two researchers will jointly conduct the first visit, and one or two researchers will conduct the second visit to Cohort 1 pilots. Visits will last two to three days depending on the scope of the pilot. No single interview will exceed two hours, and site visit interviews will average 1.25 hours.
Focus groups of participants. Focus groups will provide the study team with rich information about participants’ experiences learning about, enrolling in, and participating in the P3 program. Each Cohort 1 pilot will have an average of three focus groups, and we expect that eight youth will participate in each 60-minute focus group. Thus, in total, each Cohort 1 pilot will have up to 24 youth participating in focus groups. Each Cohort 2/3 pilot will have a single focus group with up to 8 youth participating. Focus groups will not be conducted during exemplar visits.
Survey of partner managers. The short survey will be administered to all P3 pilot managers after the conclusions of their on-site interviews to collect systematic information from respondents about their perceptions of their P3 collaboration. To the extent possible, site visitors will collect the completed survey from the respondents at the time of completion. However, if the respondent is unable to complete the survey at the conclusion of the interview, then they will be able to seal it in a pre-addressed and pre-stamped envelope to return it to the study team.
Partner network survey. The survey will target the service delivery staff within each identified partner who has the most comprehensive knowledge of service delivery decisions related to P3 within his or her own entity, and of communication about service delivery issues with other partner staff. This strategy could result in surveying a range of respondents, from a program manager of a community-based organization to the employment counselor of another organization. The study team will identify individuals who fit the criteria and will obtain contact information for them during the first round of site visits.
All data collection for the survey will take place via email. The use of electronic delivery allows for self-administration of the partner network survey, as well as tracking survey completions. We will use partner contact information gathered during the site visit to distribute the survey to the partners identified by each selected pilot. We will attach the survey as a PDF to an email introducing its purpose in the study and providing instructions for its completion and return. Partner respondents can open the PDF attachment on the introductory email, enter their responses, and forward the email back to the sender with the document attached at a convenient time for them. We plan for two additional follow-up communications with nonrespondents: (1) a follow-up email from the study team and (2) a reminder email from the study team. See Instruments 5 through 8.
This study does not require statistical methodology or estimation. We will analyze the data collected from the site visits, focus groups, and partner network survey using qualitative and descriptive methods.
The qualitative and descriptive analysis plan consists of a mixed-method approach with three steps:
Organize the qualitative data from site visits and focus groups. To manage the data effectively and systematically, the study team will develop structured templates and checklists for site visitors to use to distill the interview and focus group information they collect during site visits. Then, the study team will organize all the data from the site visits using qualitative data analysis software, such as NVivo.
Identify themes in the data within and across pilots. Once it organizes the site visit and focus groups data, the study team will examine the data to determine the effects that P3 has had on the pilot’s system or network for providing services to disconnected youth and how the provision of services to these youth has changed. Both agreements and discrepancies in respondents’ responses or across data sources can provide useful information on the pilot’s implementation experiences and successes and challenges. After analyzing and organizing all of the site visit data within each pilot, the study team will examine the data across the pilots to look for similarities in system changes and models of organization, service delivery, or other characteristics.
Analyze the partner manager survey data. The survey will explore the quality of the P3 partnerships from the partner manager perspective. We will tabulate the responses of the survey by pilot and also explore responses by partner types, for example, public and private partners, to analyze differences between them. We also will conduct simple tabulations and analyses to analyze changes in collaboration between the first and second site visits.
Conduct a network analysis using data from the partner network survey of pilot partners to develop typologies of partner relationships. The study team will use two primary measures to describe and depict service delivery networks within and across P3 grantees: density (interconnectedness) and centrality (prominence). Density is the proportion of possible relationships that are actually present, and measures the extent to which each partner is connected with all others across the network as a whole. Centrality measures the prominence of individual entities within the network. The study team will examine the measures of prominence for specific partners within the select networks across the two measures for comparison. We expect that we may find differences in the network interconnectedness and centrality of partners based on any communication and based specifically on changes in communication.
Using sociograms, the study team will illustrate the patterns in the size of partner networks, the strength of the relationships across partners, and the direction of partnerships. These sociograms will depict the density and centrality of pilot networks based on contact frequency and the change in contact frequency since becoming involved in P3. In addition to sociograms, the study will produce tables that present network-level characteristics such as overall density and centralization (measures discussed above). The study team will also present figures of helpfulness ratings in the P3 pilots, illustrating the centrality of specific partners in each network.
There are no unusual problems requiring specialized sampling procedures.
For the Cohort 1 site visit interviews and focus groups, there will be two cycles of data collection about one year apart. The first round of Cohort 1 site visits, including interviews and focus groups, will focus on the initial planning and development of the pilots and the services being delivered to youth participants. The second round of Cohort 1 site visits, which will also include interview and focus groups, will focus on the sustainability of P3-driven changes and on improvements to systems and programs for disconnected youth. During both the first and second visits, we will discuss potential efficiencies created in the P3 system, the service paths for youth, and staff experiences, among other topics.
In addition, in Fall 2018, we plan to conduct telephone interviews with each Cohort 1 program director as the P3 grant ends. This interview will focus on sustainability of activities and lessons learned. As indicated, we also will conduct site visits to up to six pilots for which P3 has engendered the most change in community systems. The purpose of these visits will be on best practices and scalability of program efforts.
We will only administer the partner network survey once following the 2018 site visits.
Site visits and phone interviews. Pilots have agreed to participate in the national evaluation as part of the grant award, but we will work with grantees and their partners to schedule site visits at a time that is convenient within a three-month site visit window for each round of visits. The first round of Cohort 1 visits was conducted during summer/fall 2017; the second round will be conducted during spring 2018. We anticipate conducting phone interviews with Cohort 1 grantee managers in Fall 2018. We expect to hold Cohort 2/3 visits in Fall 2018. We anticipate the visits to six exemplar pilots to occur in late 2018. Because the visits will involve several interviews and activities, there will be flexible scheduling to accommodate the particular needs of respondents. In addition, site visitors will meet with in-person interview respondents in their own offices or at a location of their choice.
We will use several well-proven strategies to ensure the reliability of site visit data. First, the small site visit team, all members of which are experienced site visitors, will be thoroughly trained in the issues of importance to this particular study, including how to probe for additional details to help interpret responses to interview questions. Second, this training and the use of the protocols will ensure that data collection is standardized across sites. When appropriate, the protocols will use standardized checklists to further ensure that the information is collected systematically. Finally, we will assure all interview respondents of the privacy of their responses.
Focus groups. We will ask pilot program staff for their assistance in recruiting and encouraging the participation of P3 youth for focus groups. Knowing that some youth might need additional encouragement, we will provide a $20 gift card to those youth who participate in focus groups. For Cohort 1, we are assuming three focus groups per pilot across the two visits, with about eight participants in each focus group. We expect to conduct a single focus group with about eight participants for each Cohort 2/3 grantee. Focus groups will not be conducted during exemplar visits.
Partner manager survey. To achieve high response rates, we will request that our on-site interview partner manager respondents complete the survey at the conclusion of the interview. Some respondents might not be able to complete the survey at that time. We will already have established rapport with these respondents through the interview, so will follow up with them directly to request completion of the survey. The partner manager survey will be conducted during each Cohort 1 and Cohort 2/3 visit.
Partner network survey. To encourage survey response, the study team will use methods that have been successful for numerous other Mathematica studies. We expect a 90 percent response rate for the partner network survey, based on recent Mathematica network analysis surveys such as the Evaluation of the SSI/SSDI Outreach, Access, and Recovery (SOAR) Initiative. Based on the study team’s previous experience conducting network surveys using approaches similar to those described below, we can achieve this response rate through a strategy of outreach and multiple follow-ups, timing and means of data collection, and ease of survey completion.
Outreach materials will be clear and succinct and convey the importance of the survey data collection from multiple perspectives. The initial outreach email conveying the survey to each P3 pilot partner will (1) introduce the study and its purpose; (2) highlight DOL as the study sponsor; (3) explain the voluntary nature of participation in the survey; and (4) provide a contact name, number, and email address for questions that sample members may have. A follow-up email, sent about a week after the initial invitation to complete the survey, will contain similar information about the purpose of the study and encourage the P3 partners to complete the survey. After two weeks of nonresponse, we will send a final reminder encouraging a response.
The timing of the survey shortly after a site visit will support high response rates. The survey will be administered in waves within one to two weeks after each pilot’s 2018 site visit. The study team will have made some initial contact with a majority of potential respondents during site visit interviews, lending them a level of comfort with the purpose and legitimacy of the survey. In addition, administering the survey by email can support a high response. The P3 partners are working professionals, and we expect that email will be the most effective means of communication.
Lastly, the partner network survey is designed to be easy to complete. The questions are written in clear and straightforward language, and the average time for respondents to complete the survey is estimated to be 10 minutes. The full list of partners at each pilot will be pre-loaded into the PDF document so respondents for each pilot will be able to select a response that relates to each pilot partner. The PDF will allow respondents to enter responses (only check marks or Xs are necessary) but will prevent them from revising any other text or information in the questionnaire. Respondents will be able to view the question matrix with each possible category of response (across the top) and the full range of partners (down the side) on one sheet. This approach is common among network data collection to help respondents consider their levels of connectivity with all partners of the network and assess their relationships using a common set of considerations regarding the question of interest. The approach only works when the network is known ahead of time and the number of partners is relatively small; it also has the added advantage of facilitating data entry and analysis in that respondents provide information about all partners in the network in a systematic and streamlined way. Missing responses on particular questions or for particular partners will be represented as “no” responses, that is, no communications or no change in communication or helpfulness. The default for nonresponse is that there is a minimal relationship between the partners.
We administered the partner network in one pilot in 2017 and partner manager surveys with all Cohort 1 pilots during the 2017 visits. For the partner network survey, we delivered surveys electronically by email following the site visit. The participants were asked to complete the survey and record the amount of time that it took. Following the administration, the study team debriefed on the process and determined steps to improve completion, such as asking the P3 program director to remind partners of the survey and sending follow-up emails to sample members within five days after the initial email. For the partner manager survey, we requested that interview respondents complete the survey at the conclusion of the interview. Similarly, we used the first site visit to test our site visit and focus group protocols and procedures and refined them as needed for subsequent visits.
No consultations on the statistical methods were necessary for this study. We list staff responsible for collecting and/or analyzing data in Table B.3.
Table B.3. Individuals who will collect and/or analyze data for the National Evaluation of P3
Company |
Contact |
Mathematica Policy Research P.O.
Box 2393 |
Linda Rosenberg (Project Director) Liz Clary Lindsay Read Feinberg |
Social Policy Research Associates 1330 Broadway, Suite 1426 Oakland, CA 94612 (510) 763-1499 |
Andrew Wiegand |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Dorothy Bellow |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |