Follow up Survey

Memorandum

Date: December 7, 2016

To: Janet Javar

Chief Evaluation Office, Department of Labor

From: Karen Gardiner, David Judkins, Lauren Eyster and Adrienne Smith

Subject: TACCCT Round 4 Evaluation Pretest Results

This memo summarizes the pretest findings for two TAACCCT Round 4 data collection instruments: the 12-month participant follow-up survey and new questions added to the college survey to support the employer study. It first describes the pretest procedures for each instrument. It then describes the pretest findings and proposed changes. Copies of the revised instruments (clean and tracked) were sent under separate cover.

Pretest Procedures

Abt SRBI conducted the 12-month survey pretest November 18-20. Abt worked with Ivy Tech, one of the nine outcomes study grantees, to obtain a pretest sample. The sample consisted of 31 students who enrolled in one of the two programs included in the outcomes study (ETI105 or ITSP 135) in Fall 2015 at one of the campuses not participating in the study.

Five interviewers conducted the pretest via telephone using a paper version of the survey. Interviewer training occurred on Thursday evening, November 17, 2016. The following evening the interviewers began calling the sample and continued throughout the weekend. Together they completed nine interviews by Sunday evening. Amongst the nine interviews, there was a mix of those who had stopped taking classes (five respondents) and those that were still taking classes in their program. Of those who had stopped taking classes, three had or were working. After each survey was completed, the respondent was asked to complete the debriefing survey. The debrief inquired about clarity of the introduction script; assessment of overall interview experience; how hard it was to answer the questions in the survey; how well the survey introduced new sections; whether questions could be answered by memory or by referencing information the respondent had with him/her; whether any questions were difficult to understand; and whether the time to complete the survey was about right or too long. The debrief also included section-specific questions related to understanding of training receipt, quality and educational progress questions; training-related support questions; work history and training plans questions; and household income and composition, and public benefit receipt questions. Respondents received a “thank you” payment of $35 for their time.

Abt conducted the new college survey questions pretest between November 18th and December 2nd. Abt contacted five grantees and completed the pretest with three of them: Ivy Tech, South Central Community College and Manchester Community College. The eight questions, added as a new Section H of the Round 4 college survey, inquire about the colleges’ relationships with employers. The information collected from the college survey will be used to identify employers for the employer perception sub-study. The team did not conduct a pretest of the full survey as it was pretested for an earlier OMB package.

Abt asked grantee staff to consider whether the questions make sense or if wording changes are suggested, whether grantee staff filling out the survey have the information to answer the questions, whether any questions are irrelevant or unrelated to their grant project, whether they would feel comfortable providing the names of employer partners, and whether there is anything concerning TAACCCT employer partnerships in general that isn’t included.

Pretest Findings and Recommended Changes

12-month survey

The average length for all nine interviews was 18.56 minutes. The timing of the survey was not started until the beginning of Section B (Section A was respondent verification), thus the introduction and screener section are not included in the overall timing. Abt SRBI estimates that the paper administration of the survey added about five minutes to the interview time. The added time, though, is offset by the need to add four minutes to account for the introduction and closing sections, and a minute added for each of Sections C (training-related supports) and D (employment characteristics) to account for those who would require the longer versions of these sections (that is, skip patterns resulted in shorter versions of these sections). This leads to a net increase in time of two minutes over the average length recorded in the pretest, meaning the survey length should be 19-20 minutes as currently designed.

Based on the pretest findings, the evaluation team does not recommend adding new questions or deleting questions. We do propose a number of clarifications to existing questions. Exhibit 1 shows by section the questions that were revised as a result of the pretest.

New college survey questions

Exhibit 2 summarizes the comments and proposed changes. Of the three grantees, one had no direct feedback on the questions. A second requested a clearer explanation of why the evaluation team asked for employer names. The third suggested other categories to include in Question H6 (“For each of your employer partners, what types of activities have they participated in for your TAACCCT-funded project? Check all that apply.” Other suggested changes were more appropriate for in-depth interviews rather than the standardized answers for the college survey.

The evaluation team does not anticipate these changes will add to the time burden for completing the survey.

Exhibit 1: Changes to 12-month Survey Questions

Original Question	Comment	Revision
B3. Were you awarded one or more industry-recognized training certificates, licenses, degrees or other credentials as a result of completing the required classes for the [PROGRAM NAME] at [SCHOOL NAME]	3 respondents inquired whether credentials included an Associate’s Degree	Added “degree” to list of potential credentials
B7. Using the dates you gave me, my computer is showing that you attended the [PROGRAM NAME] for [WEEKS IN PROGRAM]. How many of those weeks were you on break instead of actively attending classes, not counting breaks?	Respondents had difficulty answering the question that required ability to subtract number of weeks in class from the total in a semester or year	Revised question to ask about number of weeks on break instead of weeks attending class, which is expected to be easier to calculate
B8. How many hours per you attend classes for [PROGRAM NAME]?	Unclear how to answer for online classes and for multiple semesters	Probes added specific to how to calculate if class is online and how to average hours across semesters
B9a. How many credits have you earned, excluding credits transferred from other institutions and credits for prior learning?	Several responses that were ranges instead of a specific number and some in which numbers given exceeded the range provided for the question	Raised limit on hours and added probe if respondent gives range indicating we do not need an exact count, just give me your best recollection, please
B10a. How many transfer credits did they accept?	Same comment as B9a	Same revision as B9a
B11a. How many credits for prior learning did they award you?	Same comment as B9a	Same revision as B9a
B13. As part of your classes for [PROGRAM NAME] at [SCHOOL NAME], did you participate in work-life physical environments with appropriate tools and other equipment where you could practice your skills?	Confusion about the meaning of “work-life” physical environment	Corrected question to read “work-like” physical environment
B14. Is there a recommended program at [SCHOOL NAME] that leads to a next level credential for any students who finish [PROGRAM NAME]?	Confusion about definition of “next level credential”	Changed question to ask about a “higher-level credential” and added a probe to clarify higher level.
B15b. Do you plan on returning to college in the future?	Confusion as to whether college where program operated or any college and purpose of returning	Revised question to ask about returning to any college in the future to complete or earn another credential
B18. As part of your studies for [PROGRAM NAME] at [SCHOOL NAME], have you been offered any of the following opportunities for direct experiences with occupations related to your studies or career goals?	Unclear what sub-question “Class offered on-site at local employer” meant	Revised to indicate class taught by instructors from local employer or class offered on-site at local employer
C1. I am going to read you a list of funding sources that you might have used to pay these school or living expenses. For each of them, please tell me if the source helped pay for any of these expenses: (10 sub-questions about different funding sources)	Respondents struggled with C1d (loans in your name or the name of a family member). They seemed unclear about whether Pell grants were the same as FAFSA or whether FAFSA should be included in their responses to this question. Part of this confusion may be that there isn’t an option in this question that talks about Federal loans. Respondents also unclear if question related to one semester or more	Revised introduction to Section C to clarify answer should be for whole study period and also defined components of FAFSA (grants and federal loans). Separated loans taken by respondent and those by family/others and asked latter in new sub-question
C.4 We are also interested in the different types of advising services you have received at [SCHOOL NAME]. While taking classes for [PROGRAM NAME] have you received [SERVICE] from the school itself or through a referral from the school to any other source such as a church, community-based organization, non –profit organization, or local employment office?	Respondents wanted to respond “both”	Revised question accordingly
E6. How many of your children 18 years or younger currently live in your household?	Not all respondents have children	Revised to make clear if respondent has children, how many live in the household

Exhibit 2: Changes to College Survey Questions

Original Question/Script	Comment	Revision
Section H introduction	It is not clear why the evaluation needs this information	Edited introduction to clarify names will be used to populate a series of questions to understand how the college partnered with each employer
H6. For each of your employer partners, what types of activities have they participated in for your TAACCCT-funded project? Check all that apply.	Additional sub-categories would be informative	Added two sub-categories: (1) speaking to TAACCCT participants or engaging in other activities such as job fairs or facility tours to increase awareness of career opportunities and (2) Providing support to TAACCCT participants (e.g., scholarships, tuition assistance, time off from work)

2