Site Visit Protocol - P3 Instrument 1

Instrument 1

p3 site visit master protocol

This page has been left blank for double-sided copying.

NATIONAL EVALUATION OF P3

Master Protocol

introduction

Good afternoon [morning]. My name is __________________and I am from Mathematica Policy Research [Social Policy Research Associates]. [If two-person visit, other member introduces him/herself.] I am [We are] part of an independent research team that is studying P3 for the United States Department of Labor, in collaboration with its federal partners.

As you know, P3 is an innovative initiative designed to provide communities with flexibility across federal discretionary programs with the goals of improving the services provided to and the outcomes of disconnected youth. We are interested in learning from you about the network of organizations involved in working with these youth on behalf of P3 and the services provided. We would like to hear about your experiences working on P3 in your community and your perceptions about P3’s successes and challenges.

Everything that you say will be kept strictly private within the study team. The study report will include a list of the P3 grantees and their partners. All interview data, however, will be reported in the aggregate and, in our reports, we will not otherwise identify a specific person, grantee, or partner agency. We might identify a pilot by name or a type of organization or staff position if we identify a promising practice.

This discussion should take about <duration> minutes. Do you have any questions before we begin? Do you consent to participate in this discussion?

<If recording interview>: I would like to record our discussion so I can listen to it later when I write up my notes. No one outside the immediate team will listen to the recording. We will destroy the recording after the study is complete. If you want to say something that you do not want recorded, please let me know and I will be glad to pause the recorder. Do you have any objections to being part of this interview or to my recording our discussion?

Note for site visitors: These questions encompass the full set of questions of interest to the evaluation’s systems and implementation analyses. To the extent that information for some questions have already been collected through other sources, such as P3 conferences, telephone conversations, and documents, please reframe the questions to confirm the collected information. Similarly, please tailor other questions, as appropriate, given what is known about the pilot.

Public burden statement. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid Office of Management and Budget (OMB) control number. Public reporting burden for this collection of information is estimated to take an average of 1.5 hours per respondent, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [mailing address for DOL Office of the Assistant Secretary for Administration Management]. The OMB control number for this project is <OMB control number>; expiration date <date>.

Public burden statement. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid Office of Management and Budget (OMB) control number. Public reporting burden for this collection of information is estimated to take an average of 1.5 hours per respondent, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [mailing address for DOL Office of the Assistant Secretary for Administration Management]. The OMB control number for this project is <OMB control number>; expiration date <date>.

Public burden statement. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid Office of Management and Budget (OMB) control number. Public reporting burden for this collection of information is estimated to take an average of 1.25 hours per respondent, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to [mailing address for DOL Office of the Assistant Secretary for Administration Management]. The OMB control number for this project is <OMB control number>; expiration date <date>.

A. community context

• Please describe the current set or network of organizations that interact to provide services to disconnected youth in the community. What are the major organizations? The secondary organizations? To what extent are these organizations reflected in/participating in the P3 pilot?

• Describe the local labor market conditions related to youth (for example, youth employment, the youth demographic, youth issues such as youth crime/offenders, gang participation, high school dropout rate, youth unemployment).

• Prior to P3, what, if any, factors were thought to impede the local network’s efforts to most effectively serve disconnected youth? What external factors related to federal funding and reporting requirements impeded efforts? What external factors related to state funding and reporting requirements impeded efforts?

• What internal community factors, such as historic agency boundaries, have impeded efforts?

• Prior to P3, what, if any, factors have helped the local network’s efforts to most effectively serve disconnected youth? What external factors, such as federal or state incentives for collaboration, have improved efforts? What internal community factors, such as a history of collaboration, have aided efforts?

• Prior to P3, what aspects about the network and system for providing services to youth did you feel were inefficient or ineffective in achieving positive outcomes for youth? What did you hope to change through P3?

B. Defining the Pilots

Organizational structure/system

• What is the role of the grantee agency? Is it the lead agency for P3? Does it provide direct services for disconnected youth? For other populations?

• Which agency forms the backbone of the pilot and supports and coordinates the work of the partnership?

• Who are the partner agencies? Does P3 involve any state-level partners? Public-private partnerships?

• What does each partner bring to P3? What are the roles of each partner? Which partners provide direct services to youth under P3?

• Are any other organizations that are not actual P3 partners participating in P3 as stakeholders or in some other way? What are they? What role(s) do they fill?

• What, if any, roles does the local evaluator have in the ongoing P3 program?

Program model and stage of development

• What was the genesis or initial idea that formed the basis for the P3 application? What idea or previous experiences initially brought the organizations of the P3 collaboration together?

• What is the program’s theory of change? That is, how does the program, through its services, expect to facilitate improved outcomes for disconnected youth? [If the program has a theory of change figure, use as a basis for discussion.]

• How was the program’s theory of change developed? What partners were involved? Did different partners have different ideas about what the theory of change should be?

• Please describe the pilot’s program model. What are the key features of the pilot? [Probe for the services being provided to youth participants.]

• How is this model and the services being provided to youth different from before the pilots? For example, is the pilot offering new services? Does it expand or build upon or integrate existing services?

• What gaps in services or inefficiencies across funding streams were identified that P3 was designed to address?

• How were gaps or inefficiencies identified? Were some gaps identified that P3 is not addressing? Why is P3 not addressing those gaps?

• At what stage of development/implementation is the pilot? What, if any, factors are facilitating or hindering development (for example, staffing, partner agreements, recruiting, state and local economic and policy conditions)?

Flexibility¹

• What were the areas of flexibility that you sought as part of your application? Please describe how particular instances of inflexibility have limited the work of your organization and/or the network of organizations.

• As you developed your program—beginning with the P3 application through implementation—have partners discovered that any perceived gaps and/or needed flexibilities did not exist?

• What has been the pilot’s experience in requesting or negotiating any state flexibilities required for the pilot? What have been the areas of concerns, and how have state agencies responded?

• Has there been any need for local government or other types of agency flexibility to implement P3 as designed? Please describe.

• What has been the role of various partners in identifying and/or negotiating for any flexibilities, and with which entities and for which funding streams?

• What, if any, of the perceived gaps or needed flexibilities did not actually require any statutory changes but rather the result of differing interpretations of laws? Were these misunderstandings related to federal or state laws or regulations?

• How is P3 flexibility addressing any of the limitations on organizations’ work and provision of services to youth that the grantee and community have identified?

• What, if any, additional areas of needed flexibility—at federal or state level--have the program partners uncovered? What type of flexibility would be needed, and from whom? How were these new areas identified?

• Are there any new partners you would need to engage in order to implement these new flexibilities (such as engaging state and local government agencies)?

• What, if any, has been the role of state and local governments in allowing the grantee and its partners requested flexibility?

Funding

• What are the funding sources for the pilot’s youth services? In what ways are the different funding sources being used?

• Which funds are being braided? Which funds are being blended?

• Did the pilot’s funding plans—either during the planning stage or in the final application—involve different funds for blending and/or braiding than were included in the final performance agreement? If so, what were they and why (or what is your understanding of the why) the pilot was unable to braid or blend them?

• Has the pilot leveraged any additional funds as a result of developing the P3 system infrastructure? If so, how are these leveraged funds being used? For what time period do they have the additional funds?

• What has been the role of state agencies in working through funding for P3?

• How has the pilot used its grant funding (for example, data system building, collaboration/ communications, evaluation)? How important are these funds to the success of the pilot?

• What complications and challenges has the pilot and its partners faced in working through funding for P3? How has the pilot worked through them?

Waivers

• What are the different waivers being used and to which funding streams are they tied? How are they being used? How are they expected to result in improvements for the network of organizations or for youth?

• Were some waivers considered during the P3 application writing phase that were ultimately not requested? What were they, and why were they not included?

• Were some waivers ultimately not approved? What were they? What is your understanding of why they were not approved?

• Did you need any state or local waivers to implement your pilot? What were they, and what have been your experiences in obtaining waivers from state and/or local governments?

• Since grant award, have you requested or are you considering additional waivers? If so, what are they and what need do they address?

C. partnerships, management, and communications

Partner network

• How did the lead organization and partners come together to design the P3 project? Who brought the organizations together to respond to the P3 notice? Who took the initial lead?

• To what extent had the pilot partners worked together before the pilot and in what capacity? [Probe to understand their past relationships and how they have worked together before.]

• What was the role of each of the different partners in planning and developing the P3 application? What was each partner’s role in identifying areas for flexibility, the appropriate funding streams, and any needed waivers?

• In what ways are the P3 partnerships formalized? Do they have contracts or MOUs? If they have MOUs, have they established one MOU across all partners, or separate MOUs or agreements? What is included in the MOU(s) and/or contract(s)?

• What agreements, if any, are in place between partners regarding P3 funding? Are any agreements in place regarding the grant funding? That address resource sharing for P3 or how the cost allocation for P3 is allocated to different funding streams?

• How has the partnership worked to establish a shared vision for the pilot and serving disconnected youth?

• What factors have inhibited or enhanced the partnership? For example, have competing demands on partners’ time made it difficult to schedule meetings? Has the example set by a few led others to devote the time necessary for a strong partnership?

• In what ways has the P3 flexibility enhanced organizations’ abilities to partner and collaborate to provide these services? How has using waivers and blended and/or braided funds benefited or changed these partnerships?

• Looking back at the theory of change, how is this partnership network under P3 contributing to improved youth outcomes?

• Has the pilot established relationships between new partners and/or enhanced/changed existing relationships? How?

• What are the strengths and weaknesses of the P3 partner network? Are all the right organizations involved as partners? Are any organizations/programs missing from the partnership? Which ones and why?

Management and continuous program improvement

• How are management and policy decisions made about the pilot? Who makes these decisions? The grantee or lead agency or a joint body? What is the role of various partners in the management of the pilot and program services?

• Do the grantee and partners have a leadership team or board of its partners for the ongoing work of P3? If so, is this newly formed or adapted from a previous board? Are there standing committees? What are their roles, and who sits on the board and different committees?

• What management practices (for example, joint plans, joint meetings, reporting practices) have been developed or are under development for the pilot? How is this different from before P3?

• How is the pilot assessing its performance? What information is being collected to monitor the ongoing work of partners? What mechanisms or systems are in place to collect the information? What is the role of the various partners?

• Have shared measures for success across partners been established?

• What processes are in place to assess the performance information collected to identify possible areas for improvement? What is the role of various partners in ongoing program improvement?

Communications

• What mechanisms have been created for the partners to collect and share general information and data about the pilot (for example, meetings, data management systems, dashboards, and reports)? Note questions about individual participant data are below.

• What forms do communications take (for example, mostly phone, email, in-person)? Are there formal and informal means of communications? Please describe.

• How often do you communicate with your equivalent at other organizations partnering on the P3 pilot?

• To what extent are these communications different from what you experienced with each organization prior to P3? How are any of these mechanisms or tools for communication different or new as compared to before P3?

• To what do you attribute any differences in communication (for example, supervisor/leadership guidance, receptivity of other organization, funding supporting interactions)?

• What are the strengths and weaknesses of these information flows to date?

• What is communicated to staff of the various partners about working together across partners? Do managers encourage collaborations across their staff? How?

• What information is communicated to the larger network/community about the work of P3? How is the information communicated?

D. the p3 program

Development of P3 program/intervention

• How did the pilot partners determine the intervention(s) to be implemented as part of P3?

• What resources were used to identify evidence-based programs or other programs to be provided under P3? To what extent did the partners draw on the existing base of evidence to inform their program design?

• How was the program design developed? Which partners participated in the design? When? Were additional partners included as the P3 proposal and design took shape?

• Were there differences among partners in how the P3 program should be developed, such as the services to be provided to youth and the roles of the different partners? If so, what were they, and how were they resolved?

Staff

• What staff at the different partner/other programs are providing P3 services? What are their roles? How many FTEs is each partner contributing to the pilot? What are their qualifications?

• How are those P3 staff positions funded? Are they funded by blended or combined funding streams? By one?

• Does the pilot conduct/provide any training to staff specific to P3? Are staff of different partners provided with joint or cross-training?

• How do the frontline staff of the different partners communicate? What mechanisms are in place to facilitate their communications?

• How are these communications between partner frontline staff different under P3 than before P3?

Youth participants

• What is the target population for the P3 pilot? Why was this target population selected?

• What are the eligibility criteria for participating in the program?

• How are youth identified or recruited for the pilot? For example, are they identified or referred by school staff, are they participating in other youth-focused activities and invited to participate, or are they recruited by other means? What is the outreach and recruitment role of the different partners?

• What methods and recruitment/promotional materials have been developed to increase awareness of and interest in the program? What strategies and materials have been most effective? Least effective?

• Has the youth targeted and eligible for services changed at all from the original program design? Since implementation? How and for what reasons?

Intake and enrollment

• How, when, and where does intake or enrollment into the P3 program occur?² Who/what partner conducts intake/enrollment?

• Is intake into P3 integrated across partner programs? Does eligibility information required for any and all partner programs get collected through the P3 intake process?

• To what extent is information collected at intake for P3 shared across partners? What partners have access to this information? Are partners able to use the information collected at P3 intake for intake/enrollment into any of their non-P3 programs/service?

• What ways, if any, has the intake process into services for disconnected youth changed as a result of P3 (for example, more integrated or centralized process across providers, more sharing of information, more effort to collect common information at intake)? What are the factors that led to the changes?

• What, if any, were the challenges faced in changing the intake process? For instance, were some partners reluctant or unable to share youth information? What were the issues, and how were they resolved?

Youth services

• What services do youth receive under P3? Do these services differ by youths’ characteristics (for example, age, in school/out of school)? If so, how and why do these services differ for youth with different characteristics?

• How do the services youth receive under P3 differ from the services they would receive if not participating in P3?

• What particular services are being provided to participants by each organization?

• What is the expected length and “dosage” of participation? For example, is enrollment for one school year? Are youth expected to spend a certain amount of hours per week of enrollment in certain activities? Please explain.

• What is considered completion of the program? Are youth timed out of participation or does a particular milestone (for example, high school graduation) need to be achieved?

• Are there follow-up services? [Probe for who receives any follow-up services and what the services entail.]

E. Data systems and sharing

Context

• In general, how open are organizations within the community to data sharing and/or data integration?

• Prior to P3, to what extent did organizations providing services to youth share data? What data were shared across organizations?

• Had there been any work prior to P3 to integrate data systems and/or improve the sharing of the services across organizations? If so, what were the goals of these efforts and were they accomplished?

• Were any agreements in place across organizations/P3 partners regarding data sharing? If so, what were they?

• In the past, what challenges were encountered in sharing data across providers?

P3 data systems and sharing

• What data system(s) are used to track youth participation and outcomes? Are these data systems new or are they adaptations of existing systems?

• How are these data systems shared/used by different levels of staff at different partner organizations?

• Have data agreements between providers/partners been developed under P3? With which partners? For what data?

• What specific data—both service receipt and outcomes—are being collected about P3 participants?

• How are these data being collected? Which staff enter data into system(s)? From what partners and how? From participants? From other sources?

• What data being collected are shared across partners?

• What reports are being generated about the P3 program and its participants? How are those reports being used and by whom?

• For how long is the pilot planning to track/follow up with P3 participants? What data will they be collecting? Who will be collecting the data, and how? Through follow-up with participants, partners, and/or other administrative data sources?

• What new types of tracking, data sharing, or outcomes monitoring are resulting from P3?

• How, if at all, has P3 changed the partner network’s ability to share data?

• Did you encounter any challenges related to data sharing under P3? Were these different than challenges you may have encountered previously regarding data sharing? What were the challenges, including their source?

• Were any of the challenges addressed? If so, how? Is that a change from previous attempts? If so, what was different?

F. federal role and technical assistance

• How did your community/agency first learn about the opportunities available through P3? Had you/your organization provided earlier input into federal discussions/forums about ways to improve services provided to disconnected youth?

• What was the role of the federal P3 agencies in informing the community/P3 partners about the opportunities available through P3? How helpful were the initial webinars?

• Did you receive supports from other sources on applying for P3? What were they? Were they helpful?

• What other information about the P3 application process would have been helpful? What about the process could be strengthened?

• What was involved in the negotiations with the lead federal agency to finalize the grant award? What were some of the challenges in the process, and how could they have been addressed? What were some of the strengths of this process? How could this process be improved?

• What was your experiences in working with the federal government to finalize the grantee’s performance agreement? How did this process differ from other federal grant programs you have participated in? How did the process

• How have you continued to work with the federal agencies, including the lead agency and/or the consulting agency? Please describe your interactions with the federal agencies, both individually and any coordinated communications across the agencies. Which agencies does the grantee work with and on what issue? How have agencies communicated with you about monitoring processes, reporting requirements, etc.?

• What is your assessment of the quality, frequency, and timeliness of communications with the federal agencies?

• To what extent are your interactions with the federal agencies the same or different from interactions you have had as part of other federal grant programs? How could the interactions be improved?

• To what extent have you had interactions with other P3 pilots? How have these interactions helped/not helped you in implementation of your pilot?

• How coordinated and timely have agency responses been to any questions and requests for technical assistance? What about this process could be improved?

• Have you been satisfied with the supports your pilot has received from the federal agencies and its technical assistance providers in the ongoing implementation of the pilot? Why or why not? What has worked best? What has not worked as well?

• In what areas would you like technical assistance in the future?

G. Assessing P3

Assessing P3’s potential for local change and innovation

• In what ways has your organization been affected by its participation in P3? In what ways, if any, has it benefited? What effects, if any, have not been positive?

• In what ways has the network of organizations serving youth—both the broader network and the subset focused on P3—changed as a result of P3? What system-level improvements or changes have occurred?

• What is your assessment of how P3 has enabled local communities to change the environment for collaboration across partner agencies? Did waivers provide the community with the opportunity to combine funds and work collaboratively as expected? Why or why not?

• What is your assessment of how P3 has enabled the community to design and implement innovative services for youth? Why?

• Under P3, what factors at the local level have affected—hindered or facilitated—the systems or infrastructure for providing services to disconnected youth? What about at the state, tribal, or federal levels?

• To what extent have the experiences of the P3 pilot been known within the larger community? How have they been communicated?

• To what extent have these experiences helped inform or spur any similar types of organizational partnerships and systems change?

Service delivery and systems efficiencies

• Are there any indications of improved efficiency or reduced cost for providing services to youth? If so, what factors led to these efficiencies? What is your evidence of improved efficiencies?

• Has the pilot been able to enroll into P3 the number of youth anticipated? If not, what factors affected enrollment into the P3 program?

• Has P3 enabled the pilot to serve more youth than the individual partners/programs would have in the absence of P3? If so, how many more? Why?

Sustainability

• What are the grantee’s plans for sustaining system- and program-level improvements attained under P3?

• What is the potential to sustain P3-driven changes? What elements are likely to be sustainable? Why? What elements are likely not to be sustainable? Why?

• What are the challenges to sustaining P3’s system- and program-level changes and activities?

Perceptions of P3 and pilot’s success

• What is your assessment of how well the overall P3 initiative—at the national and local levels—achieved its goals? Why? Were some elements of P3 more successful than others? Why?

• What is your assessment of how the Federal governance structure for P3 has been implemented in practice? From the local perspective, to what extent has this been a promising model for other grant programs?

• What has been the youth response to the services offered through the pilot?

• What do you think the federal P3 agencies could do to improve the overall initiative?

• What is your assessment of how well your pilot has achieved its goals? Do you perceive that youth’s outcomes have improved as a result of P3? Why? What do you base this assessment on? What factors do you believe have contributed to success and/or challenges?

Challenges and lessons learned

• What have been the main surprises—both good and not so good—in your P3 work? Which have been encountered at the local or tribal level? At the state level? In working at the federal level?

• What are the lessons that can be drawn from the P3 pilots about developing integrated strategies for improving the outcomes of disconnected youth?

• Are there any plans for applying lessons learned beyond the P3 pilot to other services/populations within the community?

1 Note that this section asks general questions about flexibility, and the next two sections ask more specific questions about P3 funding and waivers.

2 Note that this question and others, unless otherwise specified, focus on the enrollment and services provided through the P3 program that participants receive. Generally, questions are not addressing the processes and/or services for the local evaluation.