Site visit interview guide

OMB Approval No. XXXX-XXXX

Expiration Date: XX/XX/20XX

TechHire/SWFI Site Visit Protocol

Background for OMB submission: A subset of these questions will be asked during the second round of implementation research visits to each of the five TechHire/SWFI RCT sites. Each visit is to last up to two days. Some detailed questions will not be asked in the TH sites but will be asked in the SWFI sites. Finally, not all questions will be asked of all site partners. Certain questions, for example, will be applicable to the employer partners but not to the other partners.

Introductory statement for grantee staff: The TechHire/Supporting Working Families Initiative (TH/SWFI) study is being conducted by Westat and MDRC, two research firms, under contract to the U.S. Department of Labor. As part of the study, evaluation team members will be visiting each of the sites implementing the Randomized Controlled Trial (RCT) aspect of the study twice, to understand how the TH/SWFI initiatives are being implemented, the challenges being encountered, and the lessons about implementation and operations being learned. In these visits, we will be talking with grantee directors and staff and individuals from partner organizations, including employer partners. Our aim is to learn from your experiences, not to audit or judge your procedures or programs. The views you express will be kept private, and nothing we publish in the study will identify you by name or provide enough information for others to identify you.

Introductory statement for other organizations (program partners, employers): The U.S. Department of Labor (DOL) is funding a major study of the TechHire/Strengthening Working families Initiative (TH/SWFI).Two research firms, Westat and MDRC, are conducting this study under contract to DOL. We are staff from MDRC. [Local site or program name] is participating in this study. During this site visit, we are speaking with staff from a number of organizations involved in this program, including your organization. Our purpose is to better understand how your organization interacts with the [local program name], and any challenges you have encountered or lessons you have learned regarding [local program name] so far. The views you express will be kept private, and nothing we publish in the study will identify you by name or provide enough information for others to identify you.

Confidentiality statement: I want to start by thanking you for taking the time to speak with us today. Your perspectives and insights will be very helpful to the study. We have many questions and we are going to talk with many different people during our visit, so do not feel that you need to answer every question if other staff at your organization are better able to address a topic. And, understand that your participation in this discussion is voluntary; you may choose to not answer some questions. Finally, while we will take notes over the course of our interview, so we can remember the information we collect, your responses will be kept private, and we will not share the information that you provide, in a way that could identify you, with other people at your organization, including your supervisor, peers, or supervisees.

Do you have any questions before we begin?

Interviewee Background Information – Grantee Program Managers, Program Partners, Employers

1. What is your official job title or position, and how long have you been in this role?

2. What are your primary responsibilities in your job?

3. What is your role vis a vis the [name of TH/SWFI program]? How long have you been involved with the program?

Etiology of TH/SWFI Program – Grantee Program Managers, Program Partners, Employers

1. Why did your organization apply for DOL funding for [local program name] or choose to participate in [local program name]?

2. Did you establish a new program with this DOL grant or did you expand an existing program? What components of the [local TH/SWFI program name] were in place before you received this grant?

3. What are your organization’s overall goals for the [local TH/SWFI program name]?

   1. Who originally established these goals?

   2. Have these goals changed over time? If so, how and why?

   3. Are the goals modeled on or informed by an existing program model that has been successful elsewhere?

4. How does the design of the program relate to the program’s goals?

   1. What is the purpose of each aspect of the program’s design?

   2. How did you determine which components or services to include in the design of your program?

   3. In what ways, if any, was the design tailored to the needs of specific local industries or employers?

Targeted Geographical Areas and Populations, and Economic Environment – Grantee Program Managers, Program Staff

1. What is the program’s geographic service coverage area? What defines this area?

2. Why was this area selected and proposed?

3. What are the demographics of the service coverage area (e.g., race/ethnicity, immigrants)? Have these demographics been stable for a while or have they recently changed?

4. Which types of individuals form the target population of the program?

5. Why was the target population selected?

6. Has the target population changed since the grant was awarded?

7. What are the specific individual-level eligibility requirements for the program?

8. What are the major industries and/or large employers in the service coverage area?

9. What are some of the characteristics of, in particular, the IT [add other industries] in the area? Who are the major employers in these sectors? In what ways are these sectors growing or shrinking? What types of jobs are employers most frequently seeking to fill in these industries? What are typical wages in these jobs?

10. Have any major events recently impacted the economic environment, such as big employer closings or moves?

Organizational Roles and Staffing – Grantee Program Managers, Program Partners

1. How would you describe your organization’s role in the TH/SWFI program?

2. Where is the TH/SWFI program situated within your overall organizational structure? Who has authority over the program?

3. What roles, if any, do local American Job Centers or other employment service networks play in the delivery or management of the program?

4. Besides the TH/SWFI grant, are there other sources of funding supporting the TH/SWFI program components and services?

5. What is the staffing structure of the program? How many staff are in each position? Were new staff hired specifically for the TH/SWFI program?

6. Do staff work exclusively on the TH/SWFI program or do they also have non-program responsibilities?

7. Do you feel that your program has enough staff? Have you had much staff turnover since receiving your TH/SWFI grant? Have there been any changes to the staffing structure since the start of the grant?

Partnerships – Grantee Program Managers, Program Partners

1. In your TH/SWFI program, who are your primary partners? With which of these do you have a contract or other financial arrangements to participate in your program?

2. Had you partnered with these organizations before? If so, are their responsibilities similar to their previous responsibilities in your TH/SWFI program? Is this why they were brought into the program? If not, why were the partners brought into the program?

3. Have any difficulties been encountered in the partnerships?

4. What role do employer partners play in your TH/SWFI program? Did they help to design the program? Did they assist with course/curricula development and oversight? Do they play a role in providing training or job search assistance to program participants? Do they have a hiring pipeline from the program? Do any employers have hiring quotas with the program? Are program graduates guaranteed jobs with any employers?

5. What particular services does the TH/SWFI program provide to employers?

Participant Recruitment – Grantee Program (and Partner, if applicable) Managers and Staff

1. Under the TH/SWFI grant, are the individuals you are recruiting for the TH/SWFI program any different than the individuals you have recruited for programs in the past? If so, how so? Are you conducting recruitment any differently for the TH/SWFI program than for previous programs?

2. What methods of recruitment have resulted in generating interest among the most potential program participants? What methods of recruitment have resulted in recruiting the most program-eligible potential participants?

3. Do you work with other organizations to assist with recruitment and/or provide referrals to your program? If so, which organizations typically generate the most program-eligible potential participants?

4. In marketing your program to potential participants, what do you think is your most effective “pitch”? What most commonly draws people in and motivates them to take steps towards enrollment?

5. Are there programs offered in your community that are similar to your TH/SWFI program? If so, who offers these programs and what are their names? How are they similar to and different from your TH/SWFI program? Who do they serve and what services do they offer?

6. Have you changed your recruitment strategies over the course of receiving the TH/SWFI grant? If so, why, in what ways, and what was the effect of the changes?

7. Have you encountered recruitment challenges? If so, what are the chief ones and how were they addressed?

8. Are there any recruitment “best practices” that you think would be helpful for other organizations to know if they were to try to implement a program similar to your program?

Intake and Assessment – Grantee Program (and Partner, if applicable) Managers and Staff

1. Who determines whether an interested individual is eligible to participate in your TH/SWFI program? How is eligibility established (e.g., TABE testing, program coding examples, substance abuse screens, etc.)? Has the program ever faced situations in which there are more eligible individuals (taking into account that a portion of those eligible will be assigned to the control group) than funded program “slots?” If so, what happens in that case?

2. Have eligibility criteria or methods of establishing eligibility changed over time? If so, how and why?

3. How long does the intake process take, from initial interest on the part of a potential program participant to the establishment of eligibility?

4. Once eligibility is established and random assignment has occurred, what is the next step for individuals randomly assigned to the program group? Is there an assessment process? If so, who does the assessment and what does it involve? How are the assessment results used? Are individual service or employment development plans developed? If so, what do these consist of, on what are they based, and when are they developed? When are individuals considered to be “enrolled” in the program and tracking starts for them in the DOL MIS?

5. What is the next step, following random assignment, for individuals randomly assigned to the control group? Are any services referrals provided to them? If so, how are these determined? Do staff in your organization have any further (post-random assignment) interactions with control group members? If so, what ones are typical? Under what circumstances might control group members have their activity tracked in the DOL MIS?

Training Services – Grantee Program (and Partner, if applicable) Managers and Staff

1. What type of training is most commonly provided in your TH/SWFI program? In what industry sectors is training most commonly offered? For what occupations are people most commonly trained? Has this changed over time? If so, why?

2. What are the most common modes of training? Is it classroom-based or offered on the job? Can it involve internships or apprenticeships? Has this changed over time? If so, why?

3. Is help with adult basic skills provided? If so, how and under what circumstances?

4. What is the typical sequence of activities in training? Does everyone start in the same activities? Are certain activities required of everyone? What determines when someone moves from one set of service activities to another? How long does it take a typical participant to complete the entire sequence of training services?

5. If classroom training is offered, who provides it? Is there open entry and exit? Are only TH/SWFI program participants in the classes or are non-program individuals in the classes as well? How long does it take a typical participant to complete the curriculum?

6. If training is offered on the job (such as in OJT positions, internships, apprenticeships, work experience, etc.), how are placements made? Who provides the on the job training at the placement? Are participants paid while they are in such placements? Are participants eligible for regular, unsubsidized jobs at their placement employer at the end of their training? Are there any circumstances under which such jobs are guaranteed to completing participants?

7. Do participants receive credentials, licenses, or school credits when they complete the training? If so, describe these. Are any exams or demonstrations -- other than successful completion of the training – required in order for participants to receive these credentials, licenses, or credits? Were these created for the TH/SWFI program or were these established ones? Which industry sectors and employers are most likely to know about and recognize these credentials?

8. Are participants asked to participate in other activities once they complete training? If so, which ones and for what purpose?

9. What are some of the most common reasons participants do NOT complete training?

10. Have any changes to your overall training services occurred over time? If so, what changed and why?

11. How is the offered training similar to or different than other training offered in the community? For example, is the TH/SWFI training more likely to include contextualized basic skills instruction? Proceed at a faster or slower pace? More likely to accommodate participants’ life circumstances? Offer credentials not available elsewhere? Other similarities and differences?

Child Care Services– Grantee Program (and Partner, if applicable) Managers and Staff

1. Prior to receiving the TH/SWFI grant, what were the major child care challenges encountered by parents in your area when seeking to participate in training? Which of these challenges did you seek to address with the TH/SWFI grant? Did you establish a new program with this DOL grant or did you expand an existing program? Include new partners? Expand existing systems? Hire additional staff?

2. How does the design of your grant-funded child care changes relate to your child care goals?

3. Specifically, what type of child care assistance is offered through your TH/SWFI program? Who provides this assistance? What does the assistance consist of? How, when, where, and for how long is it provided?

4. If child care assistance is not offered through your TH/SWFI program, is this assistance offered to program participants elsewhere? Does the program provide referrals to participants to access this assistance? Who provides the child care assistance and what does it consist of? For how long can it be provided?

5. Are any requirements placed on TH/SWFI program participants in order for them to be able to receive child care assistance?

6. Have any changes to your child care assistance offerings or procedures occurred over time? If so, what changed and why?

7. How is the offered TH/SWFI child care assistance similar to or different from other child care assistance offered by your organization or in the community? For example, is the TH/SWFI assistance more coordinated with participants’ training schedules? Does it have priority access to child care slots? (And, if so, typically how do individuals “bumped” due to TH/SWFI’s priority meet their child care needs?) Is it more likely to result in child care covering odd hours or providing care for very young children? More likely to accommodate participants’ life circumstances? More likely to result in licensed care? More likely to be able to continue past the training to support regular, unsubsidized employment? Other similarities and differences?

Systems Level Activities – Grantee Program and Partners

1. Prior to receiving the grant, what were the major child care challenges encountered by parents in your area when seeking to participate in training? Which of these challenges did you seek to address with the grant? How does the grant address these challenges?

2. What systems level objectives is your grant program trying to achieve? Which stakeholders or organizations are part of the system in the local area? What is your program trying to change, if anything, about the way that these local organizations work together?

3. How successful do you feel the program has been at achieving its systems level goals? What’s working as planned? Do you feel you will meet your systems change goals?

4. What has been difficult about pursuing this child care systems change?

5. [If partner] Since the [name of program] developed and began its operations, have there been changes in the systems by which low- and middle-skilled parents seeking job training access child care? Please describe.

6. [If partner] Have there been any changes in the organizations involved in training and child care?

7. [If partner] Have there been any changes in the way your organization interacts with other organizations providing training and child care?

[If partner] What are the remaining barriers to low- and middle-skilled parents accessing child care to enable them to participate in training and employment? From your perspective, how might those barriers be overcome?

Employment, Counseling, Financial, and Other Support Services – Grantee Program (and Partner, if applicable) Managers and Staff

1. What are the major non-child care support services available to participants in your TH/SWFI program? Who provides these services? What do the services consist of? How, when, where, and for how long are they provided? To what extent are these services funded through your TH/SWFI program?

2. In particular, what services or supports are available to TH/SWFI participants who complete training? Does the program help them with job search skills? If so, how? Does it connect them with job openings? If so, how? Does it have contact with individuals after they start jobs? If so, how? How long are these services generally provided? Do staff view these services as valuable? Do training completers use these services?

3. Are any requirements placed on TH/SWFI program participants in order for them to be able to receive these employment, counseling, financial, or other support services? Are these services also available to individuals not in the TH/SWFI program?

4. Have any changes to these service offerings or procedures occurred over time? If so, what changed and why?

5. How are the non-child care support services offered to TH/SWFI participants similar to or different from other support services offered by your organization or in the community?

Service Engagement – Grantee Program (and Partner, if applicable) Managers and Staff

1. What strategies do you use to assist TH/SWFI program participants in keeping up their attendance in program activities, staying engaged with the program, and completing their training?

2. How do you determine if a participant is having trouble understanding the materials or keeping up with the work? In these cases, what actions do you take?

3. What actions, if any, are taken by staff if participants have poor attendance in training or other program activities? If participants drop out of the program altogether?

4. What options are available to participants who, for various reasons, cannot complete the program? Do you refer them to other services or programs?

5. On average, how long can participants receive TH/SWFI services, relative to the point at which they are randomly assigned?

6. Do you have a sense of the extent to which program participants are obtaining regular, unsubsidized jobs in the industry sectors and occupations for which they have trained?

Specific Questions for Employer Partners – Employer Partners

1. Why did your company get involved in the grant, e.g., hire more skilled workers, increase productivity, reduce recruitment costs, etc.?

2. Did you have a prior relationship with the grantee?

3. How important are industry-recognized credentials or degrees for the types of occupations in which the grantee is training participants? Have you had difficulty filling these positions? Do you sometimes relax requirements?

4. Do you employee any graduates of the program?

   1. If no, why not? Were there any participants in paid work experience (internships, OJT) that you wanted to hire but could not?

   2. If yes, in what types of positions are the employed? Are there opportunities for advancement or additional training provided?

5. What are your impressions of the skills of program graduates? How do they compare to those of other workers, including H-1B visa holders? How do their hard and soft skills compare?

6. To what extent are the social support services provided by the program effective at alleviating barriers and helping participants to stay in jobs?

7. How has participation benefited your organization, e.g., hire more skilled workers, increase productivity, reduce recruitment costs, etc.?

8. How satisfied are you with the program?

9. To what extent do you think other (non-partner) employers are aware of the program?

10. Do you plan to collaborate with the grantee after the end of the grant period? In what ways? What are the challenges to continued collaboration?

11. Have there been any changes to the local economy during the grant period? How have these affected the ability of the program to recruit, train, and place participants?

Performance Management – Grantee Managers

1. How do you assess the performance or quality of the training that the TH/SWFI program is offering/using? Are there performance benchmarks that have been established?

2. How do you assess the performance or quality of the child care assistance and other support services that the TH/SWFI program is offering/using? Are there performance benchmarks that have been established?

3. In general, how do you track and assess the performance of your partners in providing TH/SWFI services?

Program Successes and Challenges So Far – Grantee Program (and Partner, if applicable) Managers and Staff

1. What do you believe have been the TH/SWFI program’s greatest successes so far? What are the main factors that have contributed to these successes?

2. What do you believe have been the main challenges the program has encountered – during start-up and to date? How did you seek to overcome these challenges?

3. Is there anything you would do differently if you were just starting to plan and implement your TH/SWFI program now? If another locality was interested in implementing a program very similar to your TH/SWFI program, what would you tell them to be sure to do and to NOT do?

Sustainability and Future Plans – Grantee Program (and Partner, if applicable) Managers and Staff

1. Are you planning to continue to offer this program past the current TG/SWFI grant end date? If so, are you planning on making any modifications and/or changes to the program? If not, what are the primary reasons you do not plan to continue the program?

2. What resources would be required for you to continue to operate this program past the current TH/SWFI grant end date?

3. Are there other approaches, strategies, or services that you believe would contribute to better outcomes for the population being targeted in your TH/SWFI program?