P3_OMB Package_Part B (REVISED)

Part B: Collection of Information Employing
Statistical Methods for the National Evaluation of the Performance Partnership Pilots for Disconnected Youth (P3)

September, 2016

Submitted to:

Office of Management and Budget

Submitted by:

Chief Evaluation Office

Office of the Assistant Secretary for Policy

United States Department of Labor

200 Constitution Avenue, NW

Washington, DC 20210

This page has been left blank for double-sided copying.

CONTENTS

Part B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 1

B.1. Respondent universe and sampling methods 1

1. Site visit interview respondents 3

2. Focus group participants 4

3. Partner manager survey 4

4. Partner network survey 4

B.2. Procedures for the collection of information 4

1. Data collection 4

2. Statistical methodology, estimation, and degree of accuracy 6

3. Unusual problems requiring specialized sampling procedures 7

4. Periodic data collection cycles to reduce burden 7

B.3. Methods to maximize response rates and deal with nonresponse 7

B.4. Tests of procedures or methods to be undertaken 9

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data 9

TABLES

Table B.1: Description of the nine pilots 2

Table B.2. Sample sizes by data collection activity 3

Table B.3. Individuals who will collect and/or analyze data for the National Evaluation of P3 9

This page has been left blank for double-sided copying.

Part B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

The Chief Evaluation Office (CEO) of the U.S. Department of Labor (DOL) has contracted with Mathematica Policy Research and its partner Social Policy Research Associates (hereafter “the study team”) to conduct the National Evaluation of the Performance Partnership Pilots for Disconnected Youth (P3). The evaluation will provide information to policymakers and administrators about how the nine pilots in the first cohort used the flexibility offered through P3 to pool funds and waive programmatic requirements and how this flexibility helped them overcome significant hurdles in providing effective services to and improving outcomes for disconnected youth. In addition, the evaluation represents an important opportunity to study the implementation outcomes and system changes that the pilots are able to achieve and the outcomes of and impacts on youth participants of the pilot programs.

This package requests clearance for three data collection activities conducted as part of the evaluation’s implementation and systems analysis: (1) site visit interviews; (2) focus group discussions with P3 youth participants; (3) a survey of partner managers; and (4) a survey of partner service providers or partner network survey.

B.1. Respondent universe and sampling methods

In October 2015, nine competitively awarded grantees were announced as the first P3 cohort. They received up to $700,000 in start-up funds and the flexibility to blend or braid discretionary funds from fiscal year 2014 and 2015 to improve the outcomes of disconnected youth. The first cohort grantees are located in eight states (California, Florida, Illinois, Indiana, Kentucky, Louisiana, Oklahoma, and Washington) and a federally recognized Indian tribe located in Texas. As required in the legislation authorizing P3, the grantees are serving disconnected youth, defined as low-income youth ages 14 to 24 and are either homeless, in foster care, involved in the juvenile justice system, unemployed, or not enrolled in or at risk of dropping out of school. Several grantees are serving in-school and out-of-school youth, and some are focusing on specific populations such as youth in foster care or public housing. Almost all the grantees are relying on their Workforce Innovation and Opportunity Act Title I Youth funds along with other DOL, Department of Education (ED), Department of Health and Human Services (HHS), Corporation for National and Community Service (CNCS), and Institute of Museum and Library Services (IMLS) funds. Table B.1 provides additional information about each pilot. Table B.2 provides sample sizes for the three data collection efforts.

Table B.1: Description of the nine pilots

Pilot name	Location of pilot services	Anticipated number of participants	Estimated number of partners	Target population	Brief description of intervention
Baton Rouge P3	Baton Rouge, Louisiana	84	6	14 to 24 year-olds who are 2 or more years behind in school	Youth will develop an individual success plan. Program staff will develop training activities, and will encourage youth to participate in other training and education programs provided by partners.
Best Opportunities to Shine and Succeed (BOSS)	Broward County, Florida	420	4	At-risk youth in six high schools	Students will be provided a case manager. The case managers will have a 1:35 ratio of case manager to youth. The case manager will connect each participant with “evidence-based and evidence-informed” educational, employment, and personal development services that specifically address the needs of the student in regard to graduation and post-secondary success. The BOSS program will provide intensive, comprehensive, and sustained service pathways via a coordinated approach that helps youth progress seamlessly from high school to post-secondary opportunities.
Chicago Young Parents Program (CYPP)	Chicago, Illinois	140	3	Low-income women ages 16 to 24 with at least one child younger than six	CYPP is a parent engagement, education and employment program that combines two successful, research-based program models: employment and mentoring for youth and high quality comprehensive Head Start programming for children and families. All participants receive basic Head Start services plus additional mentoring, home visits, socializations, education planning, enrichment sessions, and employment.
Indy P3	Indianapolis, Indiana	80	8	At-risk, low-income youth ages 14-24; target youth in public housing	Indy P3 will provide comprehensive, concentrated, and coordinated services for cohorts of very high risk disconnected youth. Staff members called connectors (each serving 40 youth and families at a time) will develop individual service and success plans, link participants to core service providers, and share data across programs. Partners will emulate best practices and lessons learned from evidence-based models.
Los Angeles P3 (LAP3)	Los Angeles, California	8,000	24	Youth ages 16 to 24	LA P3 is comprehensive service delivery system that coordinates and integrates the delivery of education, workforce, and social services to disconnected youth. Partner agencies and WIOA youth contractors in the city of Los Angeles provide the program services. These are existing services: the aim of LAP3 is to enhance the availability of these services through the enhanced coordination of partner agencies.
P3-OKC	Oklahoma County, Oklahoma	60-70	12	Foster youth ages 14 to 21	Youth will receive: (1) modified wraparound services more consistent with child welfare services; (2) an integrated plan of services to promote service integration and foster partnerships across nonprofit and public organizations; (3) the Check and Connect intervention designed to monitor school attendance, participation, and performance; and (4) enhanced vocational development, work, and/or career opportunities achieved through wraparound, educational options, and career aspects of students enrolled in career academies.
Seattle-King County Partnership to Reconnect	Seattle-King County, Washington	200	3	Youth ages 16 to 24	The program will have three components: (1) strategic coordination of workforce development services with the state’s unique Open Doors policy, which provides K-12 funding for reengagement programs; (2) utilizing AmeriCorps members to develop a regional outreach strategy aimed at placing the hardest to serve youth in programs that best reflect their interests and needs; and (3) advancing efforts toward a shared data system and common intake process that will enhance the coordination and targeting of services across Seattle-King County.
Southeast Kentucky Promise Zone P3	7 rural southeast Kentucky counties	1,000	3	At-risk youth ages 14 to 24	The program will include a teen pregnancy prevention program, career assessments and exploration trips, academic and career mentoring and tutoring, professional development for teachers and community members, two generations of family engagement focused on youths’ parents, and paid work experience.
Tigua Institute of Academic and Career Development Excellence	Ysleta Del Sur Pueblo tribe (Texas)	45-50	2	Tribal youth members ages 14 to 17 enrolled in two local high schools	Youth will receive group sessions of an integrated Leadership curriculum based on nation building theory and the Pueblo Revolt Timeline, which includes the Tigua lecture series to teach youth about their history, language and tribal government and the various services offered by the departments. Youth will also receive individually based wraparound services.

Sources: Grantee presentations at Annual P3 Conference in June 2016; grantee draft evaluation plans; and grant applications.

Table B.2. Sample sizes by data collection activity

Respondents	Total number of respondents over evaluation	Number of responses per respondent
Site visit interviews
Administrators and staff	135	2
Focus group discussions
Youth	216	1
Parents	108	1
Survey of partner managers	90	2
Partner network survey	90	1
Total	639	--

1. Site visit interview respondents

All nine pilots of the first cohort of P3 are included in the evaluation. We will visit each of the nine pilots twice: in early 2017 and spring 2018. During the site visits, we will interview grant and program administrators and managers of partner organizations as well as frontline staff of P3 partner agencies that are providing P3 services to youth. To the extent we can, we will interview all appropriate staff but, given constraints on the length of visits and respondents’ other responsibilities, we might need to select one or two representatives from a particular staff position from among the larger group of staff who would be suitable to interview.

2. Focus group participants

We will conduct an average of three focus groups of participants in each pilot across both rounds of site visits. In pilots providing different types of services to distinct groups of participants—for example, in-school and out-of-school youth—we will conduct separate focus groups.

Focus group participants will be a convenience sample of youth who are receiving services and are willing to participate. We will ask the pilot staff to recruit P3 program participants for the focus groups. We will request that, to the extent possible, the participating youth be diverse on characteristics such as gender, age, race/ethnicity, and length of program participation to provide different perspectives on the program.

Participants of these focus groups are not intended to be representative of experiences of all youth within a pilot and are subject to bias given staff recruitment methods and youths’ own decision whether to participate. Conducting focus group with the convenience sample is to provide additional perspectives on the P3 program that is unique from what we will collect from program staff or partners. The data collection is intended to capture a potential range of experiences within the pilots to identify themes in successes, challenges, and possible improvements, rather than a sample that is statistically representative of a larger group. Prior experience has indicated that, for providing additional insights into a research question, a convenience sample is sufficient.

3. Partner manager survey

We will administer the partner manager survey to all managers or leaders of the P3 pilot partner agencies and organizations. We will request that they complete the short survey after concluding the on-site interview. The survey will provide quantifiable information about respondents’ perceptions of the P3 collaboration.

4. Partner network survey

The survey is a brief, targeted tool to explore the strength of relationships between the key entities (partners) that oversee service delivery within the local P3 system. We expect to administer the survey to the frontline staff of all identified partners of each pilot, up to a maximum of 15 per pilot. During the first site visit to each pilot, prior to survey administration, we will work with each partner to identify the appropriate survey respondent.

B.2. Procedures for the collection of information

1. Data collection

The data sources and data collection activities for the National Evaluation of P3 are as follows.

Site visits to pilots. We will visit each pilot twice. Before each pilot’s first site visit, the study team will collect and assess the key information previously collected about the pilot, such as the pilot’s partners and the services provided to its youth participants. Part A of this submission lists the research topics that the study team will explore during site visits; the main objectives are to collect and analyze data on (1) the local systems created for P3 and (2) how P3 flexibility has resulted in different and innovative services to disconnected youth.

To capture multiple perspectives, the study team will obtain information on each topic from multiple respondents so that no single person’s opinions or responses will be assumed to be fully representative. The study team will interview both managers and frontline staff in an effort to understand not only how service delivery and administrative processes are supposed to work, but also how they actually work. These respondents will include grantee lead(s), pilot manager(s), data systems manager, partner managers, and frontline staff providing services.

The site visit master protocol (see Instrument 1) will guide on-site interviews. The specific questions and length of each on-site interview will depend on the specific respondent. Two researchers will jointly conduct the first visit, and one researcher will conduct the second. Visits will last two to three days depending on the scope of the pilot. No single interview will exceed two hours, and site visit interviews will average 1.25 hours.

Focus groups of participants. Focus groups will provide the study team with rich information about participants’ experiences learning about, enrolling in, and participating in the P3 program. Each pilot will have an average of three focus groups, and we expect that eight youth will participate in each 60-minute focus group. Thus, in total, each pilot will have up to 24 youth participating in focus groups.

Survey of partner managers. The short survey will be administered to all P3 pilot managers after the conclusions of their on-site interviews to collect systematic information from respondents about their perceptions of their P3 collaboration. To the extent possible, site visitors will collect the completed survey from the respondents at the time of completion. However, if the respondent is unable to complete the survey at the conclusion of the interview, then they will be able to seal it in a pre-addressed and pre-stamped envelope to return it to the study team.

Partner network survey. The survey will target the service delivery staff within each identified partner who has the most comprehensive knowledge of service delivery decisions related to P3 within his or her own entity, and of communication about service delivery issues with other partner staff. This strategy could result in surveying a range of respondents, from a program manager of a community-based organization to the employment counselor of another organization. The study team will identify individuals who fit the criteria and will obtain contact information for them during the first round of site visits.

All data collection for the survey will take place via email. The use of electronic delivery allows for self-administration of the partner network survey, as well as tracking survey completions. We will use partner contact information gathered during the site visit to distribute the survey to the partners identified by each selected pilot. We will attach the survey as a PDF to an email introducing its purpose in the study and providing instructions for its completion and return. Partner respondents can open the PDF attachment on the introductory email, enter their responses, and forward the email back to the sender with the document attached at a convenient time for them. We plan for two additional follow-up communications with nonrespondents: (1) a follow-up email from the study team and (2) a reminder email from the study team. See Instruments 5 through 8.

2. Statistical methodology, estimation, and degree of accuracy

This study does not require statistical methodology or estimation. We will analyze the data collected from the site visits, focus groups, and partner network survey using qualitative and descriptive methods.

The qualitative and descriptive analysis plan consists of a mixed-method approach with three steps:

• Organize the qualitative data from site visits and focus groups. To manage the data effectively and systematically, the study team will develop structured templates and checklists for site visitors to use to distill the interview and focus group information they collect during site visits. Then, the study team will organize all the data from the site visits using qualitative data analysis software, such as NVivo.

• Identify themes in the data within and across pilots. Once it organizes the site visit and focus groups data, the study team will examine the data to determine the effects that P3 has had on the pilot’s system or network for providing services to disconnected youth and how the provision of services to these youth has changed. Both agreements and discrepancies in respondents’ responses or across data sources can provide useful information on the pilot’s implementation experiences and successes and challenges. After analyzing and organizing all of the site visit data within each pilot, the study team will examine the data across the pilots to look for similarities in system changes and models of organization, service delivery, or other characteristics.

• Analyze the partner manager survey data. The survey will explore the quality of the P3 partnerships from the partner manager perspective. We will tabulate the responses of the survey by pilot and also explore responses by partner types, for example, public and private partners, to analyze differences between them. We also will conduct simple tabulations and analyses to analyze changes in collaboration between the first and second site visits.

• Conduct a network analysis using data from the partner network survey of pilot partners to develop typologies of partner relationships. The study team will use two primary measures to describe and depict service delivery networks within and across P3 grantees: density (interconnectedness) and centrality (prominence). Density is the proportion of possible relationships that are actually present, and measures the extent to which each partner is connected with all others across the network as a whole. Centrality measures the prominence of individual entities within the network. The study team will examine the measures of prominence for specific partners within the select networks across the two measures for comparison. We expect that we may find differences in the network interconnectedness and centrality of partners based on any communication and based specifically on changes in communication.

Using sociograms, the study team will illustrate the patterns in the size of partner networks, the strength of the relationships across partners, and the direction of partnerships. These sociograms will depict the density and centrality of pilot networks based on contact frequency and the change in contact frequency since becoming involved in P3. In addition to sociograms, the study will produce tables that present network-level characteristics such as overall density and centralization (measures discussed above). The study team will also present figures of helpfulness ratings in the P3 pilots, illustrating the centrality of specific partners in each network.

3. Unusual problems requiring specialized sampling procedures

There are no unusual problems requiring specialized sampling procedures.

4. Periodic data collection cycles to reduce burden

For the site visit interviews and focus groups, there will be two cycles of data collection about one year apart. The first round of site visits, including interviews and focus groups, will focus on the initial planning and development of the pilots and the services being delivered to youth participants. The second round of site visits, which will also include interview and focus groups, will focus on the sustainability of P3-driven changes and on improvements to systems and programs for disconnected youth.

We will only administer the partner network survey once following the first round of site visits.

B.3. Methods to maximize response rates and deal with nonresponse

Site visits and interviews. Pilots have agreed to participate in the national evaluation as part the grant award, but we will work with grantees and their partners to schedule site visits at a time that is convenient within a three-month site visit window for each round of visits (about January through March 2017 for round 1 and March through May 2018 for round 2). Because the visits will involve several interviews and activities, there will be flexible scheduling to accommodate the particular needs of respondents. In addition, data collectors will meet with in-person interview respondents in their own offices or at a location of their choice.

We will use several well-proven strategies to ensure the reliability of site visit data. First, the small site visit team, all members of which are experienced site visitors, will be thoroughly trained in the issues of importance to this particular study, including how to probe for additional details to help interpret responses to interview questions. Second, this training and the use of the protocols will ensure that data collection is standardized across sites. When appropriate, the protocols will use standardized checklists to further ensure that the information is collected systematically. Finally, we will assure all interview respondents of the privacy of their responses.

Focus groups. We will ask pilot program staff for their assistance in recruiting and encouraging the participation of P3 youth for focus groups. Knowing that some youth might need additional encouragement, we will provide a $20 gift card to those youth who participate in focus groups. We are assuming three focus groups per pilot across the two visits, with about eight participants in each focus group.

Partner manager survey. To achieve high response rates, we will request that our on-site interview partner manager respondents complete the survey at the conclusion of the interview. Some respondents might not be able to complete the survey at that time. We will already have established rapport with these respondents through the interview, so will follow up with them directly to request completion of the survey.

Partner network survey. To encourage survey response, the study team will use methods that have been successful for numerous other Mathematica studies. We expect a 90 percent response rate for the partner network survey, based on recent Mathematica network analysis surveys such as the Evaluation of the SSI/SSDI Outreach, Access, and Recovery (SOAR) Initiative. Based on the study team’s previous experience conducting network surveys using approaches similar to those described below, we can achieve this response rate through a strategy of outreach and multiple follow-ups, timing and means of data collection, and ease of survey completion.

Outreach materials will be clear and succinct and convey the importance of the survey data collection from multiple perspectives. The initial outreach email conveying the survey to each P3 pilot partner will (1) introduce the study and its purpose; (2) highlight DOL as the study sponsor; (3) explain the voluntary nature of participation in the survey; and (4) provide a contact name, number, and email address for questions that sample members may have. A follow-up email, sent about a week after the initial invitation to complete the survey, will contain similar information about the purpose of the study and encourage the P3 partners to complete the survey. After two weeks of nonresponse, we will send a final reminder encouraging a response.

The timing of the survey shortly after a site visit will support high response rates. The survey will be administered in waves within one to two weeks after each pilot’s site visit. The study team will have made some initial contact with a majority of potential respondents during site visit interviews, lending them a level of comfort with the purpose and legitimacy of the survey. In addition, administering the survey by email can support a high response. The P3 partners are working professionals, and we expect that email will be the most effective means of communication.

Lastly, the partner network survey is designed to be easy to complete. The questions are written in clear and straightforward language, and the average time for respondents to complete the survey is estimated to be 10 minutes. The full list of partners at each pilot will be pre-loaded into the PDF document so respondents for each pilot will be able to select a response that relates to each pilot partner. The PDF will allow respondents to enter responses (only check marks or Xs are necessary) but will prevent them from revising any other text or information in the questionnaire. Respondents will be able to view the question matrix with each possible category of response (across the top) and the full range of partners (down the side) on one sheet. This approach is common among network data collection to help respondents consider their levels of connectivity with all partners of the network and assess their relationships using a common set of considerations regarding the question of interest. The approach only works when the network is known ahead of time and the number of partners is relatively small; it also has the added advantage of facilitating data entry and analysis in that respondents provide information about all partners in the network in a systematic and streamlined way. Missing responses on particular questions or for particular partners will be represented as “no” responses, that is, no communications or no change in communication or helpfulness. The default for nonresponse is that there is a minimal relationship between the partners.

B.4. Tests of procedures or methods to be undertaken

We will pre-test the partner network and partner manager surveys with the first P3 pilot site visited, using up to eight partner entities, not to exceed nine respondents in total. We will conduct the pre-tests using the same methods as those planned for the full survey administration. For the partner network, we will deliver surveys electronically by email following the site visit. The participants will be asked to complete the survey and record the amount of time that it took. Following each pre-test, the study team will debrief with each participant using a standard debriefing protocol to determine how long the survey took, whether any words or questions were unclear or difficult to understand and answer, whether the participant thought that key partners were missing (and who those were), and how the general flow and sequencing of questions worked. For the partner manager survey, we will request that interview respondents complete the survey at the conclusion of the interview. Similarly, we will use the first site visit to test our site visit and focus group protocols and procedures and refine them as needed for subsequent visits.

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data

No consultations on the statistical methods were necessary for this study. We list staff responsible for collecting and/or analyzing data in Table B.3.

Table B.3. Individuals who will collect and/or analyze data for the National Evaluation of P3

Company	Contact
Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 (609) 799-3535	Linda Rosenberg (Project Director) Liz Clary Lindsay Read Feinberg
Social Policy Research Associates 1330 Broadway, Suite 1426 Oakland, CA 94612 (510) 763-1499	Andrew Wiegand