Site Visit Interviews

Evaluation of the Young Parents Demonstration Program, Reinstatement with Change

OMB Control Number: 1205-0494

Expires: December 2015

ATTACHMENT B:

YPD IMPLEMENTATION ANALYSIS FIELD DATA COLLECTION INTERVIEW GUIDE

YOUNG PARENTS DEMONSTRATION PROGRAM (YPDP)

SITE VISIT INTERVIEW GUIDE

Introduction

I am (we are) researchers from the Urban Institute, a private, nonprofit research organization based in Washington, DC, which conducts policy-related research on a variety of social welfare and economic issues.

This project is being conducted by the Urban Institute under contract to the U.S. Department of Labor. Our visit here today is part of an evaluation of the implementation experiences of Young Parents Demonstration Program projects. A major aim of the evaluation is to identify lessons learned from your experiences in implementing the projects under this initiative. As part of this evaluation, we are conducting site visits to each of the four YPD sites. In conducting site visits to each of the project sites, we are talking to YPD project directors and staff, as well as partner organizations. We are here to learn about your service delivery model and understand how it was implemented under the YPD grant. Our aim is to learn from your experiences, not audit or judge your programs. The views you express will be kept private, and nothing we publish in this evaluation will identify you along with the statements you make to us.

Privacy Statement: Before beginning the interview, I (we) want to thank you for agreeing to participate in the study. I (we) know that you are busy and we will try to be as focused as possible. We have many questions and are going to talk to many different people, so please do not feel as though we expect you to be able to answer every question. And, we understand that your participation in this discussion is voluntary and you may choose to not answer questions you don’t wish to.

In addition, before we start, I want to let you know that though we take notes at these interviews, information is never repeated with the name of the respondent. When we write our reports and discuss our findings, information from all interviews is compiled and presented so that no one person can be identified. We also ask that you refrain from sharing anything we discuss today with others to help us ensure your privacy and the privacy of others we are interviewing.

Do you have any questions before we begin? [Respond to questions and read Public Burden Statement listed below.]

OMB No.: 1205-0494 OMB Expiration Date: 5/31/15 Estimated Average Response Time: 45 Minutes

According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of information unless it displays a currently valid OMB control number and expiration date. Responding to this questionnaire is voluntary. Public reporting burden for this collection of information is estimated to average 45 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate to the U.S. Department of Labor, Employment and Training Administration, Office of Policy Development and Research, Room N5641, Attention: Michelle Ennis, 200 Constitution Avenue, NW, Washington, D.C. 20210. Do NOT send the completed questionnaire to this address.

A. GENERAL INFORMATION ON ORGANIZATION AND INTERVIEWEE

1. Before we begin, we’d like to get some general information on you and verify some information about your YPD grant.

   1. Your YPD project name

   2. Organization name

   3. Contact information (address, telephone, fax, e-mail)

   4. Website address

2. Obtain the following information on each respondent involved in the interview (note: request a business card from each interviewee):

   1. Name

   2. Organization

   3. Contact information (address, telephone, e-mail)

   4. Title

   5. Position/role under YPD

   6. How long the individual has been involved in YPD

3. Please provide background on your organization [note: obtain brochure/recent annual report on the organization]:

1. Type of organization

2. Organization’ budget for most recently completed program year

3. Organization’s major sources of funding (e.g., WIA, funding from federal/state/city agencies, foundations, private contributions, fee for service, etc)

4. Organization’s total # of paid staff: _______

5. When organization was established

6. Types of clients/customers served or targeted

7. Major programs/initiatives operating other than YPD – for each program (excluding YPD)

• Name of program/initiative

• Number and types of clients/customers served for most recently completed program year (unduplicated count)

• Service area for program

• Brief description of services provided

• Whether the program is linked in anyway to YPD project

• Whether program is being evaluated and type of evaluation

8. Other relevant features about the grantee organization that has affected the YPD program implementation/operations

B. BASIC GRANT INFO AND PROGRAM CONTEXT

1. Tell us quickly about the overall purpose of your YPD grant project. (We will get more information and details shortly)

2. Verify with the respondent the following background information about the YPD grant (obtained prior to the visit from the Urban Institute case file)

1. Original grant period: ___________ to ___________

2. Modified grant period (Probably will not have modified grant period, but we should check): ___________ to ___________

3. Your original YPD grant amount: $________________

4. Modified (and final) YPD grant amount (Probably will not have modified grant amount, but should check): $________________

5. Leveraged funds

(Definition: cash or in-kind contributions leveraged from strategic partners including businesses, faith-based or YPD organizations, to broaden the impact of the grant-funded project. Leveraged resources were strongly encourages in the YPD grant solicitation) (amount by source, if readily available):

6. Total project budget over the time period (with modifications, including leveraged amounts):

7. Number of YPD participants in bump-up/control groups – Goal and actual number assigned at time of the site visit

8. Date random assignment began

3. Under your YPD grant, does your institution/organization operate all grant-funded programs and activities or do you contract, have financial arrangements or have memoranda of understanding with others for some programs, activities, or other components?

1. What percentage of your YPD grant activities do you contract out to other organizations? What percentage do you operate?

2. How many and which organizations do you have a contract or other financial arrangements with to operate YPD grant activities?

3. For each contracted organization (note: request subcontract agreement)–

   1. What is total amount of the subcontract

   2. What is the subcontractor’s role

   3. Does subcontractor serve YPD control group, treatment group, or both

4. What geographic area is served by your YPD grant? Possibilities include:

1. Portion of a city/county

2. Single county

3. Multiple counties (within one state)

4. Other (regional effort)

5. What has been the economic environment in which your YPD project has operated?

a. Unemployment rate for area served (start of grant; at time of visit)

(pull from BLS ahead of visit and confirm any trends)

b. Availability of job openings in area served (generally and for particular population being served)

c. Wage rates in area served (generally and for particular population being served)

d. Other local economic conditions that may have affected the project’s ability to recruit and retain participants and training participants’ ability to find employment (e.g., in- or out-migration of major employers, major layoffs, and natural disasters)

C. PROJECT OBJECTIVES AND START-UP

1. What are the main goals of your YPD initiative? Have these goals changed over the course of your project, and if so, how and why?

2. Have you modified the SOW (in your contract with the U.S. Department of Labor) during the course of the project? If so, how and why (e.g., difficulties recruiting participants, unwillingness/inability of partner to be part of initiative, change to time period or scope of work)? (Grantees have probably not made any changes, but should check)

3. How did your YPD project start-up and early implementation go (e.g., on-time, slow, etc.)?

   1. What factors facilitated project start-up?

   2. What factors hindered project start-up?

   3. What organizations did you work most closely with during the design and start-up of your YPD grant?

D. OUTREACH, INTAKE AND ASSESSMENT [FOR YPD TRAINING]

1. Under your YPD grant, what types of youth are you specifically targeting? Possibilities include:

• Youth (pre-high school)

• Youth (high school)

• Youth (out-of-school/dropout)

• African Americans/Hispanics

• Low-income/disadvantaged

• Boys/Girls

2. How have you recruited participants? What methods have you used? Possibilities include:

• Distribution of flyers, posters or other educational/informational

• Informational websites

• Toll-free informational hotlines

• Outreach campaigns using media (e.g., TV, radio, newspaper, ads on buses/bus shelters)

• Direct mail campaigns

• Door-to-door outreach campaigns

• In-person outreach presentations in the community (e.g., K-12 schools, neighborhood centers, libraries)

• Word-of-mouth

3. Have you worked with any other organizations to get recruitments or referrals to your program? Possibilities include:

• YPD partners

• Educational institutions

• Workforce system (One-Stops)

• Community or faith-based organizations

• Courts/correctional system

• Other

4. What has been the response of the targeted population to the initiative? How many participants have been enrolled in the treatment and control groups? When is the program likely to achieve its enrollment goals for the treatment and control groups?

5. Have there been recruitment challenges? If so, what challenges have been encountered and how have each of these challenges been addressed? Please also describe any “best practices” that have been identified for recruitment. Some possible challenges include:

• Had difficulty finding eligible participants

• Many who applied did not meet program eligibility requirements

• Some of the outreach strategies didn’t result in many applicants

• Partner organizations did not provide enough referrals

• Some applicants had difficulty getting to YPD facility

• Didn’t have enough resources for recruitment

• Changing economic or other conditions in the areas where recruiting

• Other similar programs competing for the same pool of participants

• Potential participants unwilling to consent to participation

• Other, please specify.

6. What incentives (if any) have been used to encourage participation and/or retention? Possible incentives include:

   1. Financial aid

   2. Work supports

   3. Supportive services

   4. Financial stipends for completion

7. Who determines eligibility of individuals to participate in your YPD training programs, and what, if any, criteria are used to select among candidates recruited? (ask for copies of any assessment materials)

Possible assessment criteria includes:

• Be referred from other specific organizations or agencies

• Attend an orientation session

• Meet income requirements

• Meet education level requirement (e.g., high school diploma)

• Pass standardized skills assessment test (e.g., TABE, ABLE, BESI, WorkKeys)

• Pass grantee’s own customized skills assessment test

• Complete interview with program staff

• Complete a program application

• Other, please specify

8. How are the service needs of participants determined? Please take us briefly through the assessment process, noting any formal assessment tests that you use under the YPD grant (e.g., TABE, interest inventories, substance abuse screening).

9. Is an individual service strategy or employment development plan created for each participant (note: request a blank copy of the form used)?

10. Has your program included a foundational skill development/remediation program component (e.g., adult basic education or GED preparation) prior to random assignment? How the foundational skill development piece fit in with your YPD program?

11. Please describe at what point individuals are randomly assigned to the bump-up or control group. Please discuss the process of random assignment (note: review flow chart on random assignment process from the Grantee Random Assignment Manual).

E. DESCRIPTION OF EXISTING PROGRAM (FOR CONTROL GROUP)

1. What specific employment, education, training, mentoring, and parenting services have been provided for YPD control group participants (as part of the existing services)? For each major service or program component, please describe:

• Discuss specific services/activities participants receive

• Who provides the service and where are services are provided?

2. Please briefly describe the typical way in which control group participants flow through the training program (i.e., from RA, through services, and to exit and during the 18-month follow-up period)? [Note: review flow chart provided as part of the Grantee Random Assignment Manual.]

3. What post-completion services do you provide control group participants in your training programs? Possibilities include:

   1. Connections to job openings

   2. Job search services

   3. Job retention services

   4. Work supports

4. Were any program components or service elements of the original program design for control participants discontinued? If yes, which ones and why?

5. Has the program faced any challenges in establishing/maintaining services for the control group? How have these challenges been addressed?

F. DESCRIPTION OF BUMP-UP PROGRAM (FOR TREATMENT GROUP)

1. What specific employment, education, training, mentoring, and parenting services have been provided for YPD control group participants (as part of the existing services)? For each major service or program component, please describe:

• specific services/activities bump-up participants receive and how these services are distinctive from existing services

• who provides the service and where services are provided?

• if training pathway – How long does training last? What specific coursework is provided (note: request listing of courses/syllabus detailing coursework/topics covered)? When training is completed, do participants receive a degree or certification?

• If mentoring pathway: Who provides mentoring services? What are the credentials of mentors (discuss variation)? How is match made between mentor and mentee? How long does mentoring last (e.g., 6 months, year, etc.)? Is there variation in duration? How often does mentor meet in-person with participant (minimum, maximum, average)? What other types of contacts occur between mentor and mentee (telephone, email) and how often? What constitutes “completing” mentorship?

• If financial incentives pathway: discuss each specific type of financial incentive, such as milestone bonus payments, including amount of payment, what needs to be done to receive a payment, experiences to date with incentive payments (e.g., which incentives are participants achieving)

2. Please briefly describe the typical way in which bump-up group participants flow through the training program (i.e., from RA, through services, and to exit and during the 18-month follow-up period)? [Note: review flow chart provided as part of the Grantee Random Assignment Manual.]

3. Has there been attrition of treatment group participants before bump-up services are completed? If yes, how many have dropped out and when has this occurred? How does attrition for the bump-up group compare to the control group? What are the specific reasons for attrition? Has the site taken any steps to reduce attrition and, if yes, what specific steps have been taken and have they been effective.

4. What post-completion services do you provide bump-up group participants in your training programs? Possibilities include:

   1. Connections to job openings

   2. Job search services

   3. Job retention services

   4. Work supports

5. Were any program components or service elements of the original bump-up program design for treatment group participants never implemented or discontinued? If yes, which ones and why?

6. Has the program faced any challenges in establishing/maintaining services for the bump-up group participants? How have these challenges been addressed?

G. YPD PARTICIPANT CHARACTERISTICS AND EARLY OUTCOMES/ PERCEPTIONS OF IMPACTS

1. Review participant characteristics, services received, and, if available, early outcomes generated from the YPD Participant Tracking System (PTS).

2. Has the site had any problems/challenges with the PTS? If so, please discuss.

3. Is the PTS characteristics, services, and outcome data valid and reliable?

   1. Does it accurately reflect number and types of clients served in the treatment and control groups? If not, why not? Discuss patterns observed in YPD participant characteristics.

   2. Does it accurately reflect types of services received by the treatment and control groups? If not, why not? Discuss patterns observed in YPD services received.

   3. Does it accurately reflect employment, earnings, and educational outcomes for the treatment and control groups? If not, why not? Discuss patterns observed in YPD services received.

4. Although it is still early, have you noticed any effects or “impacts” of the program (i.e., are there any noticeable differences between outcomes for control and treatment group participants)? If so, what are they and can you explain the differences you have observed?

5. Beside the employment and earnings outcomes maintained in the PTS, are there other important outcomes that you are tracking or think should be track for participants? If yes, please identify these other outcomes and discuss any findings to date on these other outcomes.

6. Overall, to date, what have been the greatest impacts of the YPD project on bump-up group participants? Possibilities include:

   1. Employment

   2. Self-sufficiency

   3. Skill level

   4. Self-esteem

7. Are there ways in which the program has so far fallen short of its goals for training or assisting participants? If yes, how?

8. Are there other approaches, strategies, or services that you believe would contribute to better outcomes for YPD program participants?

H. PROJECT MANAGEMENT AND STAFFING

1. Please describe your organizational structure and YPD project staffing (request organizational charts pertaining to the structure of the YPD grant)

• Type and number of project staff

• Any new hires for the YPD project

• Employed vs. contracted staff and oversight

• Location of staff

• Experience and/or credentials

• Frequency of turnover and recruitment process for new staff

• Use of volunteers or interns

2. What kinds of training/staff development activities have been provided for program staff? Please describe the extent and types of training/staff development activities, including who has conducted the training. Are there areas in which you feel there should have been more staff development/training? If yes, what are those areas?

3. Has the technical assistance provided by the DOL and the technical assistance contractor contributed to effective project implementation? Please explain.

I. PROJECT COSTS/EXPENDITURES

1. What are the major ongoing costs/expenditures for the program (note: if available, collect line item budget and line item expenditure report, e.g., breaking down total expenditures under the YPD grant for items such as project staff, rent, equipment purchase or rental, subcontracts, etc.)?

2. How do the types of participants served affect costs? What types of participants are most/least costly to serve? [Note: if readily available, collect information on per-participant costs.]

3. What has been the pattern of grantee expenditures? Have expenditures been higher during certain phases of the project? Has the project been fully implemented and reached a “steady-state” level of expenditure?

4. What kinds of resources has your YPD project leveraged?

   1. Cash

   2. Equipment/facilities

   3. Instructors

   4. Paid training for employees

   5. Internships

K. POST-GRANT PLANS/SUSTAINABILITY

1. Have you developed sustainability plans for the YPD program once federal funding is exhausted under the demonstration? If so, please describe these plans. Do you feel that your sustainability plan and leveraged resources are sufficient to sustain the activities of the grant after the completion of the grant?

2. What sources of funding are likely to be used to sustain the project or activities conducted under the YPD project?

L. PROJECT REPLICABILITY AND LESSONS LEARNED

1. To what extent do you think your program could be replicated in other localities?

2. What features of the YPD project are most amenable to replication?

3. What features of project are least amenable to replication? How does location, the target population served, or other distinctive features of your program make it either non-transferable or limit transferability?

4. To date, what do you consider your most important accomplishments under the YPD grant?

5. To date, what do you believe to be the main lessons learned from your YPD grant?

M. CHECKLIST OF ITEMS TO COLLECT FROM SITE (IF AVAILABLE)

• Background information about the locality

• Background information about the organization

• Additional documentation/reports detailing major services (especially training provided under the YPD grant)

• Diagram showing how participants flow through the program

• Organizational chart for the program

Attachment B-1