In Depth Interviews, Partners

Job Corps Process Study

Appendix E. Condensed Site Visit Protocol

SITE VISIT INFORMATION

Purpose

As part of the Job Corps Process Study, the team led by IMPAQ International, LLC, an independent social science research and survey firm, along with subcontractors Battelle Memorial Institute and Decision Information Resources, Inc., will visit 16 Job Corps centers for interviews with management and staff, observations, and focus groups with students. These site visits will serve as an indispensable information collection activity to gather and document center practices and formulate hypotheses about their relationships with performance.

The interview protocols below are designed for use in site visits to collect information from management and staff of 16 purposively selected Job Corps center directors about their programmatic and operational practices. Researchers will use information from these site visits and a follow-up survey to be administered to all 125 Job Corps centers to identify distinctive practices used in Job Corps centers and gauge their associations with various measures of center performance in attaining student outcomes.

Prior to conducting a site visit to a Job Corps center, the researchers will request a comprehensive set of documentation on it from the appropriate Job Corps regional office, including the Career Development System Services plan and reports of Federal monitoring visits. The precise set of questions asked on a given visit will be informed in part by the contents of these documents.

In addition to interviewing key staff, the researchers will conduct focus groups with students and make observations of center life through touring the facilities; protocols for both are attached in addition to the interview protocols. Researchers will note the physical layout of the center, any informational materials and notes, and locations of residential spaces, career and technical training and academic learning buildings.

The protocol below is our best representation of the set of questions that may be asked. In practice, research team members may modify the wording and omit or add certain items, depending on the context.

The protocol will be tested in a pilot visit to one of the 16 centers, and modifications may ensue to enhance the comprehensibility of questions and optimally elicit the desired information.

This appendix contains a condensed version of the site visit protocol, in which all questions are listed and the applicable respondent categories are noted in a column.

We anticipate that individual respondent interviews will proceed in approximately the following order:

1. Section A: Requests information about each respondent and her experience in Job Corps. These questions will be asked during the scheduling and logistic arrangements phase before the site visits themselves, to maximize the value of time spent on center collecting data.

2. Section B: Requests information regarding general management, operational, instructional or service delivery practices.

3. Section C: Requests information on the center leadership’s use of the Outcome Measurement System (OMS).

4. Section D: Requests information on student services and overall personal development on center.

5. Section E: Requests information on the career technical training (CTT) instruction and practices.

6. Section F: Requests information on academic instruction and practices.

7. Section G: Requests information on student-focused management practices.

8. Section H: Requests information on staff-focused management practices.

9. Section I: Requests information on organizational capacity.

10. Section J: Requests information on partner organizations.

11. Section K: Conclusion and thanks for participation.

Center Management and Staff Interviewed

Respondents who represent a broad range of center staff will be asked to share their perspectives through individual semi-structured interviews. In addition, the researchers may consult Federal oversight staff at Job Corps regional offices, executives in the organizations operating each center under contract, representatives of community partners, and managers of organizations with contracts for ancillary services at each center including outreach and admissions (OA) and career transition (placement) services (CTS).

Center staff will be interviewed individually during two-day site visits to Job Corps centers. Community partners, organizational partners, operator executives, and the Regional Office Project Manager will be interviewed over the phone prior to or after the two-day site visit, and questions for those semi-structured interviews are also included in the attached protocols.

Planned Respondents for each center include the following:

1. CD Center Director

2. AIM Academic Instruction Manager

3. CIM CTT Manager

4. WBL Work-Based Learning Coordinator

5. CPP Career Preparation Period Manager

6. CoM Counseling Manager

7. PLC Peer Leadership Coordinator

8. SDM Social Development Manager

9. CSO Center Safety Officer

10. AIn Academic Instructor

11. CIn CTT Instructor

12. RA Residential Advisor

13. SrS Senior Administrative Staff

14. BCL Business And Community Liaison

15. OAC Organizational Partners: Outreach & Admissions and Career Transition Services

16. CP Community Partner, Other than Employer

17. EP Employer Partner

18. RPM Regional Office Project Manager

Information Collected Prior to the Visit

In order to be as efficient as possible, the following information will be solicited from each planned respondent in advance of the site visits:

A. Respondent Information	Respondent Categories
What is your current (title(s)? Please briefly describe what you do.	All
How long have you worked at this center?	All
How long have you worked in the Job Corps program?	All
What other roles have you had in Job Corps or other youth development or education settings, for how long, and where?	All
Please tell us about your educational background and any professional credentials, awards, or society memberships you may hold.	All

CONDENSED SITE VISIT PROTOCOL

Introduction/Purpose of the Study

Thank you for agreeing to speak with us today. My name is <name> and this is <name>, and we are researchers from IMPAQ International, a public policy research organization based in the Washington, D.C. area. IMPAQ International and its subcontractors, Battelle Memorial Institute and Decision Information Resources, are under contract with the U.S. Department of Labor, Employment and Training Administration to explore associations between Job Corps center practices and student outcomes. ETA hopes to gain useful information for its future programmatic development and for technical assistance purposes to help the program as a whole get even better at serving the students. As part of the study we’re conducting site visits to 16 Job Corps centers. In each site, we’ll be speaking with key center staff, such as you, as well as with representatives of the operating organization and selected community partners.

We are here today to listen, observe and learn so that we can get a better, more nuanced understanding of the practices and policies your center has in place in such areas as management, student life, staffing, career development, academic and CTT instruction, and career transition. We are here to get your ideas about what makes a center succeed or fail. Needless to say, we are not going to audit or assess your work, and this visit is entirely distinct from any monitoring visits with which you may be familiar. Your responses will be kept private within the limits of the law and will not be shared with anyone outside our research team. Only a general summary of this visit without names or other references to any specific individuals will be presented to our project officer in ETA’s evaluation unit. This project will culminate in a final report that will synthesize and aggregate responses from different centers and individuals and will not attribute statements to any particular individual.

We could not conduct this research without approval from the Office of Management and Budget (OMB Control No. 1205-0XXX) and the expiration date for the approval is XX/XX/XXXX. Your response is required to obtain or retain benefits (20 CFR 670.970 and 670.975). The time required to complete these questions is estimated to average 1 hour per response.

We have a lot of ground to cover in our time together. Given this limited time, I may have to push the interview along.

<Name> will be taking notes during our interview, but if it is OK with you, I would also like to tape record our session to help me remember what you said. Do we have your consent to record this interview? (Turn on tape recorder unless the respondent objects).

B. General Questions

What are some key attributes one needs to be successful in your job?	All
How do you define success in what you do?	All
Tell us about your interactions with students. Probe: Frequency, formal v informal, scheduled/unscheduled.	All
What is the relationship like between center management and students? Probe: How often and on what occasions do senior managers – including the center director – usually interact with students? Describe these interactions.	AIM, CIM, WBL, CPP, CoM, PLC, SDM, CSO, AIn, CIn, RA, SrS, BCL
Do you think the center has a mission? If so, what is it? How is it conveyed to staff and students?	All
What have been some of the major policy, organizational, and operational emphases in recent years? Have you found any to be challenging to implement? Why? What would you suggest be done differently concerning that initiative?	All
C. Leadership and Use of the Outcome Measurement System (OMS)
Discuss the strengths, shortcomings, and characteristics of the center’s leadership. (Prompts if necessary: “Leadership” here is meant as both the individuals in the center’s administration and the attribute of influencing others and setting the tone. We want to hear your thoughts on both of those aspects).	All
How — and how frequently — do you use the OMS performance results? Probe: For what purpose(s)?	CD AIM CIM WBL CPP CoM PLC SDM AIn CIn RA SrS OAC RPM
Which OMS measures do you think are the most important? Why?	CD AIM CIM WBL CPP CoM PLC SDM AIn CIn RA SrS OAC RPM
Are there measures not included in the OMS that you use to monitor the center’s progress?	CD SrS RPM
Is information provided to management and line staff members showing the relationship between their/your daily activities and the center’s OMS performance?	CD AIM CIM AIn CIn
D. Organization of Student Services
What pre-arrival activities does your center offer for new students? (Prompts if necessary: welcome letter, brochure describing the center, invitation to tour the center, telephone call to welcome the student, email communication, other)	SDM RA OAC
Do students receive an orientation to the center when they arrive (not CPP)? What form does it take? How long is it? How frequent are the orientations? Do they include breakout components?	CD SDM RA CPP
What reports, if any, from incoming students' Admissions Counselors (AC's) do staff review? Probe 1: Who reviews them? When do they review them? Why do they review them? Probe 2: What do they do with the information? Are there particular things they pay attention to (e.g., developmental needs, background attributes), and if so, why?	AIM CIM SDM RA
Discuss the structure and courses/activities in which students engage during the Career Preparation Period. What topics are covered? Which staff are involved in these activities?	CD CIM CPP EP WBL
How does the center coordinate its services with One-Stop Career Centers and other outside providers of services for youth? What criteria are generally considered, and what categories of students are affected the most?	CD CIM
Discuss the available social development and recreational activities. What determines their selection and scheduling? How do they vary across the periods of the Job Corps program? What efforts, if any, does the center’s staff make to match such activities with the students’ interests and needs?	CPP SDM RA CP PLC
Discuss student meetings with counselors. How common are they? Are certain meetings required and why?	CoM PLC
What are the typical qualifications and backgrounds of counselors?	CoM
How has your center handled the challenge of Absent Without Leave (AWOL) issues?	CD CoM PLC SDM

E. CTT Practices
How were CTT programs chosen? Probe 1: How does the center determine the mix of CTT programs provided in- house and through national training contractors, and their slot counts? Probe 2: What role do the national and regional offices of DOL/ETA play? Probe 3: What are the practices to ensure consistency across the programs and other quality control procedures, if any?	SrS RPM
How frequently does your center update its CTT offerings? What procedure does it follow?	SrS RPM
What role does the industry council play? The employer community more generally? Probe: Contribution(s), frequency, mode?	CD CIM BCL SrS RPM
Discuss the qualifications of the CTT instructors. How does the center ensure that they are satisfactory, and what standards are in place in this regard?	CIM
Discuss the manner and timing of students’ (a) assignment to, and (b) commencement of CTT programs.	CPP CIM CIn OAC RPM
Are there waiting lists for any CTT programs? If so, how do you manage them?	CPP CIM OAC
How has this center addressed the increased focus on students attaining industry recognized credentials?	CD CIM CIn RPM
What initiatives has the center engaged in to address the National Office’s focus on Standards Based Education?	CD CIM AIM
How does the center monitor the performance of CTT programs? Probe 1: Do you use any “in-house” tools (separate from OMS), developed either here at the center or by your operator? Probe 2: Are there any problems with progress, retention, etc., and how are those addressed?	CIM CIn
What techniques ensure student progress in a self-paced environment?	CIM BCL CIn
Discuss why the center has Off-Center Training (OCT), Advanced Training (AT), and/or Advanced Career Training (ACT), if any?	CD CIM CIn RPM
What Advanced Training (AT) and Advanced Career Training (ACT) opportunities have students gone on to (at other centers)? How common is such participation? How are students selected for AT and ACT opportunities? Why?	CD CIM AIM RPM
How does the center address the needs of students who do not meet minimum academic requirements for their desired CTT program?	CIM AIM CIN AIn
What applied learning strategies do you use on your center? How and how extensively are work-based and project-based learning used?	AIM CIM AIn CIn WBL
Discuss any initiatives to provide realistic previews of employment settings in a given trade. How common are they? Why are these chosen? Are they valuable?	CIM CIn WBL EP
F. Academic Instruction
What is the center’s general “philosophy” regarding academic instruction? Probe 1: Is the emphasis on discovery, credential attainment, integration with CTT, etc.?) Probe 2: How is this philosophy conveyed to staff? To students? Probe 3: Has it changed over time, and, if so, what were the factors influencing the change Probe 3a: Personnel changes, national office guidance, operator transition, other?	CD AIM AIn SrS RPM
Are all students placed in academic offerings? Why or why not? Probe 1: If not, what categories of students are placed? Probe 2: Is a student’s past school performance considered?	CD AIM CPP
How are the curricula developed? Probe 1: To what extent is curricular development a responsibility of instructors? Probe 2: Does the center use corporate-provided or other standard “off-the-shelf” materials? Why or why not?	AIM AIn
How is content delivered? Discuss class size in different courses as well as any use of information technology in the instructional delivery. What are the strengths and weaknesses of the delivery mechanisms in place? What changes are planned and/or would be desirable? Probe 1: Content delivery mechanisms—lecturing, computers, textbooks, other?	AIM AIn
What methods does the center have in place to measure the success, impact, or effectiveness of academic instruction? Probe 1: How does it collect, use, and distribute this information? Probe 2: Are measures customized for groups of students (e.g., those in the TABE reading and math gains pools, those working towards a high school diploma/GED)?	CD AIM AIn
Does the center support integrated academic/CTT instruction? If so, in what ways? Probes: By providing staff planning time, resources?	CD AIM CIM AIn CIn
What courses and other approaches does the center have in place to prepare students for the attainment of a high school diploma or a GED certificate? How are candidates for attainment of these credentials identified? Discuss partnerships with public schools and other institutions or entities. Are there any state restrictions/regulations for GED/ high school diploma that affect student attainment?	AIM AIn
What determines whether a given student is steered toward a high school diploma or a GED certificate? Has the emphasis on high school diploma and GED attainment changed over the last several years Probe: What factors have influenced any change?	AIM AIn
What opportunities for tutoring or additional academic enrichment are available on center?	AIM AIn PLC CoM RA
How does the center prepare students for entry to colleges or other post-secondary institutions?	AIM CoM OAC
Does your center offer any other academic activities you would like to discuss?	AIM AIn CoM

G. Student-Focused Management Practices
Are the tools provided for case management useful to you? Do the case management tools meet your needs? Do you think the tools could be improved upon?	CPP CoM AIM CIM SDM CSO PLC RA
How are progress and performance evaluation panels configured on center?	CIM AIM CoM SDM RA
Have you implemented any approaches or practices that you believe help the center maintain student motivation and prevent boredom? If so, please describe.	CD AIM CIM AIn CIn SDM PLC CoM RA
What does the center do to impact the student culture? Probe: Is there something your center does that you feel especially has an impact?	CD CPP CoM SDM PLC RA
Do students engage in enrichment activities outside the center (e.g., field trips, cultural outings, volunteer opportunities)? If so, what type of activities are offered, who goes, and how often?	CIM CoM SDM CIn PLC CoM BCL RA CP
What practices do staff members utilize to engage and retain non-residential students at your center?	CD CPP CoM SDM PLC RA
What activities or approaches are used in the student dormitories to promote a comfortable and respectful environment?	SDM PLC RA
What opportunities exist for student leadership and initiative? Probe: Both within and outside the Student Government Association	CD CoM SDM CSO PLC RA
What opportunities exist for student peer-to-peer learning? Probe: Is this a formal or informal structure?	AIM AIn CIM CIn CoM PLC SDM RA
Are there any practices geared towards serving 16 and 17 year olds as a specific group on this center? If so, what are those practices? Probe: How has this changed over time; what alternatives have been explored, if any?	CD AIM CIM CPP CoM SDM PLC OAC RPM

H. Staff-focused Management Practices
Beyond technical qualifications, what characteristics and experience do you look for? Probe: Have there been changes in hiring practices over the last several years?	CD
What are the staff evaluation procedures? Probe: How are staff evaluations linked to OMS or any other performance metrics?	CD AIM CIM CoM SDM CPP
How would you characterize the organizational and staff culture of the center? Probe 1: Is it participatory? Authoritarian? Data-driven? Inclusive? Probe 2: Innovative v. structured, distributed autonomy v. hierarchy, collegiality v. everyone-for-themselves, teamwork v. self-sufficiency.	All
How is the working environment here different from other centers? If you haven’t worked at another center, what about other places where you have worked? Probe working environment: How would you rate staff empowerment versus centralization of authority on this center?	CD AIM CIM CPP CoM SDM CSO AIn CIn RA SrS WBL BCL
If you could change anything(s) about this center, what would it (they) be?	CD AIM CIM CPP CoM SDM CSO AIn CIn RA SrS WBL BCL
How do the center and operator leadership maintain contact with staff and managers (e.g., through weekly meetings, reports, surprise visits)?	CD CIM AIM CSO
How does management attempt to create a sense of common purpose, camaraderie, commitment to mission? Are these efforts successful? How could they be improved?	All
What are staff career advancement practices? Is it typical to have career ladders and promotion opportunities? How does this vary across staff categories?	CD AIM CIM CoM SDM AIn CIn RA
What opportunities are there for training and development?	CD AIM CIM CoM SDM CSO RA SrS
What kind of opportunities, if any, do staff members have to share their experiences with their counterparts at other centers?	CD AIM CIM CPP CoM PLC SDM CSO AIn CIn RA SrS WBL BCL
I. Organizational Capacity
What management practices and information does the operator use to evaluate center performance? What continuous improvement measures, rewards, and sanctions does the operator use?	CD SrS RPM
How much does the operator's expertise and experience contribute to the center's performance?	CD SrS RPM
What guidance and technical assistance does the operator provide? Is it helpful? How so?	RPM CD AIM CIM SDM
In what ways does the operator’s financial capacity influence its ability to deliver the required components of the Job Corps program?	RPM
J. Partner Organizations and Community Resources
How well do you think the OA partner(s) prepare(s) students for life on a Job Corps center? Probe 1: Do students come in with a good understanding of: Zero Tolerance, AWOL and attendance polices, center culture and dorm life, and student standards of conduct? Probe 2: What could the OA partner(s) do to better prepare students for life on a Job Corps center?	CD SrS AIM CIM CoM OAC RPM
How — and how well — does the OA partner match students to the center, and how could that be improved? Probe: Gangs, other issues?	CD SrS AIM CIM CoM OAC RPM
Could the center’s relationship with OA partner(s) be improved, for better student outcomes? If so, how?	CD SrS AIM CIM CoM OAC RPM
Could the center’s relationship with CTS partner(s) be improved, for better student outcomes? If so, how?	CD SrS AIM CIM WBL CoM BCL OAC RPM
How does the center engage with its community? Probe 1: How does it use local and community resources? What could be improved? Probe 2: To what extent are there tensions, what impact have they had, and how have they been addressed? Probe 3: Are there community champions of the center (e.g., community-based organizations, elected officials)? How does the center cultivate relationships with them and what tangible results have they delivered?	CD SrS SDM WBL CoM BCL CP RPM
Could your center’s performance be improved if there were better communication and/or coordination between any of the following entities: national office of Job Corps, regional office(s) of Job Corps, your center, your operator, your center’s OA partner(s), and your center’s CTS partner(s)?	CD SrS AIM CIM OAC RPM
K. Conclusion and Thanks
What are the biggest challenges you face in achieving desired outcomes on center?	CD
What are the top three things that you have implemented in the last year on center that you think have had the biggest impact on improving your center?	CD
Is there a best practice program on center or innovative process that you have implemented that you think is unique to Job Corps and you are particularly proud of?	CD
Are there other things you think should be considered in determining what makes Job Corps centers successful?	All

Thank you for taking the time to talk with us today.