Interview Protocol - Community College Staff

Appendix C.5. Site Visit Protocol for Community College Staff

National Evaluation of the Trade Adjustment Act Community College Career Training (TAACCCT) Program

Interview Protocol for Community College Staff

INTERVIEW INFORMATION:
GRANTEE:
PROGRAM:
ORGANIZATION/INSTITUTION:
INTERVIEW DATE/TIME:
RESPONDENT:	ADDITIONAL RESPONDENT:
TITLE:	TITLE:
STUDY ID:	STUDY ID:
INTERVIEWER(S):

Interview Topics

1. General Background Information

2. Mission and Vision

3. Strategic Alignment and Partnerships

4. State/Local Context

5. Targeted Industries

6. Planning

7. Employer Engagement

8. Recruitment and Outreach

9. TAACCCT Project Development and Implementation

10. Implementation Facilitators and Challenges

11. Progress Monitoring

12. Outcomes

13. Sustainment

14. Replicability and Lessons Learned

Introduction/Purpose of the Study

Thank you for agreeing to participate in this interview today.

My name is , and I’m a researcher from [Abt Associates, located in Bethesda, MD; The Urban Institute, a non- profit research organization located in Washington, DC; NORC at the University of Chicago, located in Bethesda, MD; Capital Research Corporation, located in Arlington, VA]. With me today is [name and affiliation]. I’m here today because Abt Associates and its partners, on behalf of the United States Department of Labor (DOL), is conducting a national evaluation of the Trade Adjustment Act Community College Career Training (TAACCCT) Program and studying the implementation of selected grantees and their capacity building efforts. This is a five-year study that includes analysis of grantee third party evaluation findings.

Your institution/organization is one of 9 sites we are visiting to conduct in-person interviews with Community College leaders, TAACCCT grant directors, a range of community college staff (i.e., faculty and instructors' advisors, transition coordinators, support staff, institutional research or data staff, technology staff, curriculum developers); employers, industry associations, and unions; technical assistance providers; community and faith-based organizations; and social service agencies.

This interview will cover the mission and vision of each institution/organization. We will also address the strategic alignment and partnerships of the TAACCCT project. The state/local context of the TAACCCT project will be explored as well as how specific industry sectors were targeted. The interview will address the planning activities for TAACCCT project implementation, how employers were engaged, and the recruitment and outreach strategies in place to reach potential students. With regard to TAACCCT project development and implementation, we will ask about the type of model/intervention that was implemented (and multiple types if applicable). For each model/intervention, we will ask about the replication and adaptation process. Since the TAACCCT grants focus on community college capacity building, we will explore a number of topics, including staff training and new hires, facilities and infrastructure, curricula development, needs assessment, supports and services, transferability and articulation agreements, and funding and leveraged resources. We will also address facilitators and challenges associated with implementation. Progress monitoring strategies and concerns will be addressed. Early outcomes will be identified and verified with program data from the third-party evaluations. The interview will conclude with some questions about sustainment, replicability and lessons learned.

These interviews will provide important qualitative information on the implementation of the TAACCCT grants across a range of institutions/organizations, including single sites, single state consortia, and multi-state consortia.

We will use what we learn from these interviews to contribute to a report on TAACCCT implementation and capacity building. This report will be submitted to DOL to inform the federal government of the progress and outcomes of the Round IV TAACCCT grants.

Confidentiality Statement [Interviewer must read this]

Before beginning the interview, I (we) want to thank you for agreeing to participate in this study and remind you that your participation is voluntary. I (we) know that you are busy and will try to be as brief as possible. We have many questions and are going to talk to many different people, so please do not feel as though we expect you to be able to answer every question. The interview today should last about 75 minutes. This interview is not part of an audit or a compliance review. We are interested in learning about your ideas, experiences, and opinions about TAACCCT implementation. There are no right or wrong answers. We want to know what you think.

In addition, before we start, I want to let you know that although we will take notes during these interviews, information is never repeated with the name of the respondent in any reports or in any discussions with supervisors, colleagues, or DOL. When we write our reports and discuss our findings, information from all the people we speak with is compiled and presented so that no one person can be identified.

To help me accurately report on the information you share, it would help if I could tape record this call; of course, I could turn off the recorder at any point if you so desire. I do not have a recorder on now; is it okay with you for me to turn it on?”

Do you have any questions before we begin?

Do I have you permission to begin the interview?

1. GENERAL BACKGROUND INFORMATION

1. Note: This section to be filled out in advance of the interview based on program documentation. Obtain the following information on each respondent involved in the interview (Note: request a business card from each interviewee):

   1. Name

   2. Institution/organization

   3. Contact information (address, telephone, e-mail)

   4. Title

   5. Position/role under TAACCCT

   6. How long the individual has been involved in TAACCCT

2. MISSION and VISION

Note: Prior to the interview review the grantee’s theory of change.

1. How does the mission of your institution/organization relate to the TAACCCT project?

2. What are the goals and intended outcomes of the TAACCCT project?

3. To what extent and in what ways does the TAACCCT project build upon or enhance your institution/organization’s strategic plan?

3. STRATEGIC ALIGNMENT AND PARTNERSHIPS

1. How are the public workforce system, educational institutions, employers and industry, and supportive services involved in the TAACCCT project?

2. What partners are involved in implementing this model/intervention? What are their roles?

Probe: Community Colleges; Public workforce investment system; Human service systems; Employer and employment agencies; Unions; Educational institutions; Faith-based and community-initiatives; Service providers; Economic Development Agencies; State Employment Security Office; Other.

3. What partners do you work with most frequently?

4. What coordination and communication mechanisms are in place for planning and implementation of the project activities with your partners? Who leads these activities?

Probe: Is this process working well? Could it be improved? How?

5. Overall, how is the partnership functioning?

   1. What are the benefits of this partnership?

   2. Have you experienced any challenges with the partnership? Or coordinating across systems? If yes, please describe.

4. STATE/LOCAL CONTEXT

1. The geographic service area for the [Single site (local/region or statewide); Consortia (multi-state)] TAACCCT project is _____________________________.? What are the major sources of employment in the area(s) where the TAACCCT-funded program is being offered?

Note: Obtain geographic area and major employer information in advance.

2. From your perspective, describe the local economic environment in which the TAACCCT project operates.

Note: Obtain unemployment rates, job market, and wage rates in advance of the site visit.

1. Unemployment rate for area served (at time of grant application)

2. Availability of job openings in area served (particularly within targeted sector)

3. Wage rates in area served (particularly for occupations in the targeted sectors)

1. Are there any local economic conditions that may have affected the design and implementation of the TAACCCT project?

Probe: In- or out-migration of major employers; Major layoffs; Base closings; Effects of recession; State/local budget cuts; Demographic changes; Other.

5. TARGETED INDUSTRIES

1. What industries and occupations are targeted through the TAACCCT project?

Probe: Aerospace; Agriculture and Food Systems; Energy-related; Engineering; Health Professions; Information Technology; Manufacturing; Natural Resource Management; Public Safety; Skilled Trades; Transit and Transportation; STEM; Other.

2. Why was this sector targeted? Please explain. If multiple sectors are targeted ask about each one.

6. PLANNING

For Community Colleges only:

1. During the planning phase, did you seek input from TAACCCT project partners about this model/intervention (e.g., other community colleges, employers, industry associates, technical assistance contractors, unions, non-profit or for-profit, community- and faith-based organizations, social services agencies, other)?

   1. If so, how was this done?

   2. What partners were involved in the planning process? What were their roles? What kinds of issues were addressed?

2. If not addressed above, ask: Did you engage with industry associates and employers to identify the skill sets, aptitudes, interests, and characteristics of workers that employers are seeking for this sector? If so, please describe this process and how you applied what was learned.

3. What activities do you/does your institution/organization engage in to align training with business needs?

Probe: Work with employers to identify specific job and career opportunities; Identify/develop courses and credentials linked to jobs and career paths; Work with employers to inform curriculum design; Work with employers to build networks for job referral; Work with employers to develop and coordinate internship programs; Work with employers to obtain and coordinate mentors in the industry; Customized pre-employment training;

4. Were there any challenges experienced during the planning phase? If so, what were they and how were they resolved?

7. EMPLOYER ENGAGEMENT ¹

1. What are the needs of employers in this area? What workforce challenges have been experienced?

2. Is there an employer engagement strategy in place? If yes, please describe.

Probe: Gather information about labor market trends (local wisdom); Align courses to local industry cluster and high-growth, high demand occupations; Establish and update curricula; Assess the adequacy of training curriculum; Solicit advice on program effectiveness; Encourage businesses to serve on advisory committees.

8. RECRUITMENT AND OUTREACH

Target Population

1. Are you involved with identifying TAA-eligible workers for recruitment?

   1. If so, what do you do?

2. What (other) types of individuals were targeted for the TAACCCT project?

Probe: Long-term unemployed; Veterans² and eligible spouses; Specific subpopulations; Dislocated workers; Entry-level workers; Individuals already in the industry; Other.

Outreach and Recruitment

3. What types of students were targeted for training using the model/intervention implemented

4. From what geographic area(s) were students recruited?

5. Did your institution/organization play a role in recruitment and outreach?

   1. If so, what did you do?

   2. What recruitment and outreach strategies and methods were used? (e.g., social media, TV/radio announcements, newspaper ads, distribution of brochures, community events, word-of-mouth referrals, other)?

   3. Had you used these methods of outreach before TAACCCT? Did the outreach and recruitment activities build your institution/organization’s capacity in any way?

6. Was the outreach and recruitment strategy coordinated across partners?

7. What was the response of the targeted population to the outreach?

   1. Was it easy to recruit participants into [model/intervention implemented]?

   2. Were there more applicants than the [model/intervention implemented] could serve (e.g., was there a waiting list to get into the program)?

   3. If there were difficulties in recruiting the number of participants originally projected under the TAACCCT project, what factors made recruitment difficult?

8. What incentives (if any) were used to encourage participation?

9. Who determined eligibility to participate in [model/intervention implemented], and what, if any, criteria were used to select among candidates recruited?

Student Population

10. What are the characteristics of the enrolled students in the TAACCCT-funded programs?

Probe: Older students; First-generation students; Students who are not academically prepared; Students who work and have family responsibilities; Students who are self-supporting and have low-incomes; Veterans.

1. Have you served/worked with this population before?

2. What populations do you typically serve?

11. Describe barriers to education/training and employment for the population served.

12. What types of services have been put in place to address these barriers?

9. TAACCCT PROJECT DEVELOPMENT AND IMPLEMENTATION

Type(s) of Model/Intervention(s) Implemented

1. According to the grant application, the TAACCCT-funded project(s) is/are implementing the following evidence-based model/intervention for the industry sector(s).

2. Note: Refer to list below and identify the model/intervention. If multiple models/interventions are being implemented, cover each one separately with question 2.

	Accelerated learning models/interventions
	Modularized learning
	Stacked and latticed credentials
	Integrated Basic Education and Skills training (I-BEST)
	Contextualized learning
	Team teaching
	Self-paced learning
	Prior learning assessments
	Other:
	Technology-enabled learning
	Simulations
	Assessment technology
	Online teaching/learning
	Real-time online interactions
	Other:
	Career Pathways
	Other

3. Let’s talk about [model/intervention].

   1. Why was this model/intervention chosen? Were other models/interventions considered? If so, which ones?

   2. Did the evidence for this model/intervention show relative advantage over other models/interventions? What are the gaps?

   3. Is this model/intervention being used for a particular industry sector? Or for multiple industry sectors?

   4. Please describe how the model/intervention implemented is aligned with competencies, skills, and industry requirements and certifications. Note: This section will require a document review prior to the on-site data collection.

      1. Competencies/occupational skills to be developed (i.e., skill requirements of the target occupation)

      2. Basic skills/ foundational skills necessary

      3. Curriculum components

      4. Duration and pace of training program

      5. Program site or location

      6. Number and qualifications of instructional staff

      7. Equipment needed

      8. If learning modalities are industry-specific

      9. Practicum, internship or employment activities

      10. Certification process (industry or employer certificate; college degree)

      11. How certifications, certificates, and diplomas can be stacked to allow building a portfolio of credentials

      12. Transfer and articulation processes in place

Replication and Adaptation

4. What is the evidence base for this model/intervention? Note: Review grant application in advance for this information.

5. Is the model/intervention a replication of a previously tested model/intervention?

6. Was the model/intervention adapted or modifications made? What was the reason for doing this?

Probe: Local conditions; Employer needs or preferences; Industry sector recommendations; Union concerns; Existing infrastructure; Partnerships; Demographic changes.

7. Was the model/intervention adapted in any way to be culturally relevant? If yes, provide specific examples of what was changed or adapted.

8. Did the lead entity request a waiver for any required program elements? If so, why was this necessary?

Capacity Building

We are interested in learning about the capacity-building activities that [name of institution/organization] has engaged in to implement the [evidence-based models/interventions noted above] (e.g., stacked and latticed credentials, technology-enabled learning).

Staffing, New Hires, and Training

9. Were there staffing changes made to implement the TAACCCT project?

1. Were administrative or instructional staff hired to implement the model(s)/intervention(s)? Or were existing staff reassigned?

2. How many staff are involved in implementing the [model/intervention implemented]? What are their roles and responsibilities?

3. Are new staff positions permanent? Are they full time or part time hires? Or are they hired solely to implement the [name of model/intervention] for the life of the TAACCCT project?

10. Was training conducted to orient staff on the [model/intervention implemented]? Please describe.

Facilities and Infrastructure

11. Have there been improvements to the community colleges’ facilities or infrastructure resulting from the implementation of the [model/intervention implemented]? Please describe.

Probe: Access to instructional supports including computer labs, tutors, and student support; Investment in technology; Technology integration; Dedicated training or classroom space; Offsite facilities; Other.

12. Have any TAACCCT project partners been involved in facility or infrastructure development or improvement? Please describe.

13. What resources have been leveraged to develop or improve facilities or infrastructure?

Curricula

14. Were investments made in developing new curricula for the [model/intervention implemented]?

15. What new curricula have been developed?

16. Will the curricula be sustained?

Needs Assessment

17. When a student enrolls in [model/intervention implemented], how are educational readiness or service needs identified?

1. Please describe the assessment process, noting any formal assessment tests that you used under the TAACCCT project (e.g., needs assessment, TABE, interest inventories, substance abuse screening).

18. Was an individual service strategy or employment development plan created for each participant? Request a blank copy of the form used.

19. Is this a new process established as a result of the TAACCCT project? Will it be sustained?

Supports and Services

20. Have academic and non-academic student supports and services been established as a result of the TAACCCT project? If so, please describe:

1. Financial stability supports ³

Probe: Assistance accessing traditional financial aid; Assistance using nontraditional sources of aid to pay for education and training; Funding for child care; Funding for living expenses (e.g., rent, utilities, food, transportation, health care); Financial emergencies (e.g., car repairs).

2. Academic supports ⁴

Probe: Tutoring; Academic advising (e.g., course planning); Academic preparation and course support (e.g., study skills).

3. Personal supports⁵

Probe: Mentoring; Case management; Mental health, substance abuse or other counseling; Targeted, small group interventions; Peer-group supports provided; Life coaches; Group support (e.g., group counseling, learning communities).

4. Sources of public support used to ensure financial stability

Probe: SNAP; WIC; TANF; Medicaid; CHIP; Section 8.

5. Career transition supports ⁶

Probe: Career navigation; Job search and placement; Assistance gaining work experience, internship search, and placement.

21. How do you expose students to information about workplace culture, including industry attendance policies and on-the-job behavioral norms?

22. For any of the supports or services described, is this for the training period only? Does the intensity of support decrease with the length of enrollment? Are “bridge programs” (i.e., post-training or intervention services or supports) provided when a student completes the course of study to facilitate his/her transition to further education or employment?

23. For each new type of support provided (financial, academic, personal, public, career transition) what resources have been leveraged?

24. Do you use information from industry and employers to develop the services provided in tandem with the model/intervention implemented?

25. What internal partnerships or relationships have been formed to implement the support or service?

1. Will this relationship be sustained?

2. Will the support or service be sustained?

26. What external partnerships have been formed to provide this support or service?

1. What institution/organizations provide these supports or services?

2. Has your institution/organization worked with the provider/entity before?

3. Is there an MOU in place?

4. Will the supports or services be provided for the life of the TAACCCT project?

27. What activities do you/does your institution/organization engage in to assist students to choose, enter and advance along a career pathway? Is this a new activity/strategy resulting from the TAACCCT project?

Probe: Applicant assessment relative to specific job/industry requirements and conditions; Identify/develop interim work experience opportunities to complement education; Student/graduate job search assistance; Industry-appropriate resume development; Industry appropriate networking tools and opportunities; Student job placement and follow-up career navigation; Counseling/supports to help students remain in jobs; Incumbent worker training.

Transferability and Articulation (for credit and credentials)

28. What transfer policies were in place prior to the TAACCCT project?

29. What criteria, policies, and procedures are now in place to enable students to transfer academic and technical courses from a two-year institution to a four-year institution for TAACCCT-funded courses and credentials?

30. What articulation policies were in place prior to the TAACCCT project?

31. What agreements are now in place to facilitate bridging non-credit courses and credentials from one institution to another among the TAACCCT community colleges or other institutions?

1. What institutions are involved?

Probe: Board of Regents; The office of career-technical & adult education of the state Department of Education; Public secondary/adult career-technical education institutions; State-supported institutions of higher education.

32. Are general education courses guaranteed to transfer to any two- or four-year public institutions in the state? Within the TAACCCT project consortia?

33. Do articulated courses adhere to recognized industry standards?

34. Have there been any institutional or logistical barriers to the transfer or articulation process? How were these resolved?

Funding and Leveraged Resources

35. What funding sources support the implementation of this model/intervention?

Probe: Federal state local public programs; Philanthropic gifts and investments.

36. Are federal funds used to provide support services? If so, which ones? See list below. How are these funds used?

Probe: Workforce Investment Act (WIA), Title I (adults and dislocated workers); WIA Title I, Youth; WIA Title II (Adult Education and Family Literacy Act); Pell Grants; Carl D. Perkins Career and Technical Training; TRIO, Student Supportive Services Program; TANF; Community Services Block Grant; Supplemental Nutrition Assistance Program Employment and Training (SNAP E&T); Community Development Block Grant (CDBG).

37. What resources were leveraged to develop and implement the model/intervention at this institution?

38. Are any in-kind resources contributed? Probe: Local volunteers; AmeriCorps members

10. IMPLEMENTATION FACILITATORS AND CHALLENEGES

1. How did TAACCCT project start-up and early implementation go (e.g., on-time, slow)?

2. What factors facilitated or helped project start-up?

3. What challenges did you experience during TAACCCT project start-up?

Probe: Have you experienced challenges associated with the [model/intervention implemented] in any of the following areas? If yes, please describe. How did your institution/organization resolve these issues?

	Challenge	Resolution
	Implementation
	Partnership
	Recruiting and hiring faculty and instructional staff for technical training in the industry sectors targeted
	Leveraging resources to implement the TAACCCT project
	Implementing any of the core components of the TAACCCT project? (Note: Review each component)
	Implementation of the evidence-based models/interventions
	Stacked and latticed credentials
	Use of online and technology-enabled learning
	Transfer and articulation policies/agreements between partner institutions
	Strategic alignment between partners and service systems
	Alignment with Previously-Funded TAACCCT projects
	Obtaining the supplies and equipment/systems necessary for high-tech instructional needs
	Using on-line and technology enabled learning
	Recruitment and retention of students
	Implementing, expanding or providing supportive services to support student retention, completion, and transition to employment
	Financial stability supports
	Academic supports
	Personal supports
	Career transition supports
	Staff turnover associated with the TAACCCT project. Probe: Did this have an effect on program implementation?
	Developing and executing articulation agreements. Probe: Within state? Among consortia partners?
	Data collection across multiple states
	Needed guidance
	Other

11. PROGRESS MONITORING

1. How does your institution/organization track key outputs and outcomes of the TAACCCT project?

2. How are the results of the TAACCCT-funded project shared with potential students and the general public?

3. To date, has your institution/organization faced any challenges regarding the data that you are required to collect and report for the TAACCCT grant?

Probe: What have they been (and with which data)?

12. OUTCOMES

Note: Before the interview ask the grantee to provide the third-party outcome data necessary to complete this section. If this is not possible, request available data/reports on TAACCCT performance/outcomes during the site visit.

1. What student outcomes can be attributed to the TAACCCT project?

1. Increased student educational attainment (certificates, degrees)?

2. Increased student attainment of Industry Recognized Credentials?

3. Increased student attainment of degrees in high-demand occupations?

4. Increased graduation rates?

5. New or strengthened relationships with employers?

2. Has the provision of new academic and non-academic supports enabled the community college to:

   1. Serve more students?

   2. Serve students from diverse communities?

   3. Facilitate student recruitment?

   4. Increase student retention?

3. Has participation in the TAACCCT grant initiative increased or influenced your institution/organization’s aspirations, strategy, organizational skills, systems and infrastructure? [See list]

• Aspirations (Mission, Vision, Goals)

• Strategy (Goals/performance targets, Program relevance and integration, Program growth and replication, New program development, Funding model/intervention )

• Organizational Skills (Performance measurement and benchmarking, Strategic planning, Financial planning/budgeting, Operational planning, Human Resource planning, Fund-raising, Revenue generation, Partnership alliance and generation, Local community presence and involvement, Public relations and marketing, Role in policymaking, Organizational development, Staffing capacity, Mission development and strategic planning)

• Systems and Infrastructure (Planning, Decision making framework, Financial operations management, Human resources management – management recruiting, development, and retention, Physical infrastructure, Technological infrastructure)

If so, please provide a concrete example. Request available data, reports, and other materials related to these changes.

4. Has participation in the TAACCCT project initiative increased or influenced the strategic alignment of the:

   1. Community college and industry and employers?

   2. Community college and public workforce development system?

   3. Community college and other education institutions?

   4. Community college and local providers?

If so, please provide a concrete example. Request available data, reports, and other materials related to these changes.

13. SUSTAINMENT

1. We have some questions about the sustainability of the TAACCCT components implemented.

   1. Which components of the TAACCCT project does your institution plan to sustain after the end of the grant? Note: Align with activities referenced in CAPACITY BUILDING section.

      1. Education and training programs overall

      2. New learning strategies/curricula

      3. Stacked/latticed credentials

      4. Technology and online learning

      5. Transfer and articulation agreements

      6. Partnerships within and outside the institution

   2. Will you sustain these specific aspects at the same level or will you scale any of these activities? Within your institution? To other institutions? Note: These probes may overlap with replicability.

   3. Will [name of entity] need to find new funding once the grant ends?

2. Are there some components of the TAACCCT project that will be difficult to sustain due to changes in technology or innovation in the field of study? Employer demand?

3. How does your institution/organization’s plan for sustainability align with your vision of the TAACCCT project?

4. How does the sustainability plan align with the strategic plan of the [name of institution/organization]?

14. PROJECT REPLICABILITY AND LESSONS LEARNED

1. To what extent do you think your program could be replicated in other departments within the college? In other localities?

   1. What features of the TAACCCT project are most amenable to replication?

   2. What features of the project are least amenable to replication? How does location, the target population served, or other distinctive features of your program make it either non-transferable or limit transferability?

2. Based on your experience with the TAACCCT project, are you likely to partner again with [name of lead entity; name of subgrantee(s)]?

3. To date, what do you consider your most important accomplishments under the TAACCCT project?

4. What do you believe to be the main lessons learned from your TAACCCT project?

5. Is there any topic or issue you expected us to cover that we have not brought up yet? (If YES): Please describe the issue(s) and explain why you think it is/they are important.

Thank you for your participation.

1 Employer engagement required for Round IV grantees.

2 Veterans receive first priority for Round IV grant programs, with TAA-eligible workers given second priority.

3 Financial stability supports fall outside the traditional system of financial aid. The purpose of this support is to ensure that students can meet the financial demands of participating in a career pathway program.

4 Academic supports help students meet the academic requirements of career pathway programs. These supports help students plan for a program of study and its requirements and provide them with the study skills and course support needed to succeed.

5 Personal supports assist participants in addressing non-financial, non-academic barriers to academic progress so that they can fully engage in their learning.

6 Career preparation helps participants use the knowledge and skills they acquire in their training to plan for a career path and secure employment.