ELS 2002 3rd Follow-up 2012 Field Test and Batch Tracing

Appendix 2
Cognitive Labs Report Summary

Forms A, B, and C: Cognitive Testing Items
ELS:2002/12 Questionnaire

Research Support Services

September 27, 2010

This report summarizes findings of the cognitive testing conducted by Research Support Services for the ELS:2002/12 study. The cognitive interviews were conducted in Evanston, Illinois, between August 18 and September 21, 2010. Eleven interviews were conducted with Form A, 9 with Form B, and 5 each with the two versions of Form C.

The following table displays respondent characteristics.

Respondent Characteristics

R#	Age	Gender	Race-Ethnicity	Form Tested		R#	Age	Gender	Race-Ethnicity	Form Tested
1	26	Female	Asian/ White	Form C—v1		16	25	Female	White	Form B
2	28	Male	White	Form A		17	25	Male	Black or Afr Amer	Form A
3	24	Female	Black or Afr Amer	Form C—v2		18	26	Female	White	Form A
4	27	Female	White/ Other	Form A		19	25	Male	White	Form C—v1
5	26	Female	Black or Afr Amer	Form B		20	27	Male	White	Form A
6	25	Male	Hispanic White	Form B		21	26	Male	Black or Afr Amer	Form C—v2
7	27	Male	White	Form B		22	24	Male	White	Form A
8	25	Male	White	Form C—v1		23	26	Male	White	Form C—v1
9	26	Female	Hispanic White	Form A		24	27	Male	Hispanic (wrote in for race)	Form B
10	28	Female	White	Form B		25	25	Female	White	Form A
11	27	Female	Hispanic White	Form B		26	28	Male	White	Form C—v2
12	25	Female	Black/ White	Form B		27	27	Female	White	Form A
13	27	Male	Black or Afr Amer	Form C—v2		28	24	Female	White	Form A
14	25	Female	White	Form A		29	25	Female	Hispanic (wrote in for race)	Form C—v2
15	27	Male	Hispanic (wrote in Latino)	Form B		30	27	Male	Hispanic White	Form C—v1

An advertisement asking for interested candidates for the interviews was posted on Craigslist on the Internet. Callers were screened using the screener in a separate appendix. The respondents interviewed had the following age distribution: age 24 (3 respondents), age 25 (9 respondents), age 26 (6 respondents), age 27 (9 respondents), and age 28 (3). Fifteen were women and 15 were men. As for race/ethnicity, one respondent self-identified as Asian, six as African-American, seven as Hispanic, and 16 as White.

Findings from Testing Form A¹

Please answer the following questions thinking about your most recent job in the past 12 months. Circle one answer for each question, indicating how strongly you agree or disagree with each statement.

Respondents had some difficulty with the verb tense of the questions. These are in present tense, but the respondents answering this form were all currently not working. They had all had a job within the past 12 months, and were answering about that most recent job. Several recommended changing the verb tense for the questions.

The jobs they were thinking of were:

#2: Roofer—had shorter jobs later but decided not to report on them. However, kept them in mind for some questions

#4: (did not report on most recent: outside sales for security company) previous job: substitute school teacher—Flip flopped between both jobs in responses and sometimes averaged answers.

#9: HR generalist

#14: telephone operator at incoming call center answering questions of filling out FAFSA forms

#17: nursing assistant

#18: Graphic designer at printing company

#20: Accountant on location at a national park

#22: Car salesman

#25: executive administrator of operations

#27: Clinical assistant at Planned Parenthood

#28: Staffing agency administrative assistant

Questions about Your Most Recent Job

1. I’m confident that I can perform my job at a very high level of skill

All Rs were able to answer this question. They circled Agree or Strongly Agree. Respondents had different interpretations for ‘very high level of skill’ during probing. However, when first answering, they were answering about doing their job well, successfully, competently.

#2: High level of skill I would refer to as efficiency. Cause certain jobs require a little bit more know-how, a little bit more knowledge, as to what you’re doing.

#4: a high level of proficiency, the ability to do the job.

#9: that you’re excelling at your work, as opposed to just being average or good.

#14: It means that I’m very good at my job.

#17: Do a bit more than just like just getting by, just like average. So actually being a little bit more doing more, without being asked.

#18: For R this meant the ability to do more than one task at a time. To do multiple tasks efficiently and creatively.

#20: I think just being able to do every aspect of the job by yourself.

#22: Well, I was thinking mainly about my ability to serve the customer and sell a car for my dealership, which is a pretty independent process, but if you use your, you know, your management the right way, and the tools that you have, it’s pretty—it can be a simple task.

#25: I am confident that I can perform my job at a very high level of skill, that you could perform your job successfully, and could apply your knowledge to that job, and be successful in that career.

#27: Well I guess high level of skill to me would be like higher education, like having a graduate degree or a Masters.

#28: When I think of high level, I think experience. I think the amount of work you’re capable of doing. I mean it all depends on what the job is. …I thought about knowledge more, like certain computer programs, like how well do you know Excel, that you know how to use every function, like that kind of thing, you know, have the skills.

2. I’m certain that I can solve big problems that occur at work

Generally the question worked well. Respondents varied in their answers. Half answered Strongly Agree, four Agree, and two Neither Agree nor Disagree.

In their narratives, Rs varied in what they considered ‘a big problem’, and that was very much tied to the specifics of the job. A common theme was that of the level of independence they had at work to solve the problems themselves or to have to get someone who could solve them. Some felt less confident because they had to ask help, but others answered Strongly Agree, because they knew what steps to take for someone else to solve the problem.

Some focused on more common, every day problems (students fight in school, problems with someone’s paycheck not being correct), whereas others thought of really BIG problems (a fire at work; a patient having cardiac arrest).

3. I’m confident that I can reach the goals I set for myself at work

Two answered Agree, nine answered Strongly Agree

Most Rs claimed to actually set goals for themselves at work, some more short term, like daily goals, and others more long term. Some Rs in jobs were goals were set by others for them, focused on that, and did not ‘hear’ the part about ‘goals I set for myself at work’. (car sales, outside sales, shift plan for nurse). Other’s goals were more about attitude: one R’s goal in her very boring job was “to stay in a good mood, despite being yelled at most of the time” (#14), another’s goal was to be accurate and responsive (#20). Those who spoke of daily goals were really mostly describing a to-do list. Most everyone felt they spoke from experience: they had set goals for themselves and reached them.

4. I’m certain that I can do my work well despite time pressures

Respondents did not give as high responses as in the prior questions. 4 Strongly Agreed, 3 Agreed, 2 Neither Agreed nor Disagreed, and one Disagreed. They did not all feel they do equally well under time pressures. The question worked well: Rs spoke of coping with deadlines, and gave examples of what time pressures they had at their jobs. Some felt a little uneasy about answering without knowing particulars, as there could be unreasonable requests that made it impossible to do a job well if the time was too short. In some cases, Rs felt the job was designed to follow extreme time pressures that did not allow them to do the job well (call center)

5. I’m confident that I can do my work well even when I need to juggle work with non-work responsibilities (e.g., in my family or community)

Question worked very well. Rs focused on the idea of not letting outside things interfere with the performance of the job. Those Rs who do not have family near them, tended to answer about work vs. non-work things, such as doing laundry, seeing friends, or taking care of the place where they live.

Occupation Interests

1. I am really interested in my work.

2 Strongly Agreed, 6 Agreed, 1 between Agreed and Neither, 1 Neither, and 1 Disagreed

For those Rs who seem to have found their field of interest, this question had positive responses, despite the fact that they are not able at the time to find a job in that line of work. Others had no investment in their most recent job, so they were not as enthusiastic.

For those who have had multiple jobs, this question made them compare in their minds how interesting they found this job as opposed to other jobs they had before. Asking this question retrospectively elicits their perception now, looking back in perspective. The question may differ from how they would have answered while they were working.

#2: “it always depends on what I’m doing. The roofing job I wasn’t necessarily interested in it, but the crew that I was working with kind of made up for it, so to speak.”

#22: Answered Agree, but not Strongly Agree, because although at the time he was working he was very interested in the car sales job, he now does not want to do that type of work.

#28: R who answered between Agreed and Neither: “I mean it got a little rough and tedious at times at a time when I just felt like I wanted more responsibility. And you know, I learned a lot about recruiting, and how the industry works, and I found it very interesting. And that’s something I look at wanting to maybe do as a career is be a recruiter, so in that sense, I was interested in the whole staffing concept. It was just the work I was doing. I wanted to feel like I existed more, or had, like I said, more roles”

2. I often get totally absorbed in my job tasks.

Question worked generally well. Probed on meaning of ‘totally absorbed’. A few thought question asked if work consumed their lives (#20)

3 Strongly Agreed, 6 Agreed, and 2 Disagreed (call center operator and national park accountant)

#2: I guess just your mind is in the job. You have forgotten about other problems because everyone wants to solve problems but when you get to a job you have to forget about all of your other problems and focus on the problems at hand.

#4: Enveloped, you know, in that same level of concentration where if you were a horse, you would have blinders on.

#9: Maybe like consumed whereas all you’re doing or thinking about, so that there’s little room for other things.

#14: This has never happened to me at a job, and I just graduated college last year. I assume they’re talking about like when you’re in a class and you find the subject really interesting, and you’re really focused on it, and you’re even thinking about it when you’re not working on it because it’s so fascinating to you, and you just want to know more about it.

#17: I would guess you’re really involved in what you’re doing. You really care about what you’re doing. When you’re at work, everything else outside that’s going on is on the back burner till you leave your job for the day.

#18: I feel like the job I’ve had, the days go by pretty quickly. And even if somebody comes up to me and starts talking to me, I get startled, because I’m too into what I’m doing. #20: I was thinking people that talk about their work outside of work, and think about their work, or I was thinking when you’re at work, you’re all about your job and you can’t focus.

#22: I mean I don’t have a perfect attention span, but I’m pretty focused. I like to do my job right. I like to do it 100-percent.

#25: When I’m doing my job, I am like a chainsaw. You don’t let anything distract you, and you get the job done. You stay focused.

#27: it’s like you really like what you’re doing and you were stuck with it, or not stuck with it but you are stuck on it; whereas, for me, it was just really busy.

#28: Nothing else around you matters. But sometimes, that’s not always a good thing, because when because when you have to multitask you have some…

3. I rarely get bored when I am doing my job.

The phrasing of this item as ‘rarely’ threw off at least some respondents. To disagree that something rarely happens is cognitively difficult, like a double negative.

Choice of response 3 was not for lack of opinion but rather trying to average the boring and the non-boring aspects of the job.

1 Strongly agreed, 4 Agreed, 3 neither agreed or disagreed, 2 disagreed, and 1 strongly disagreed

#2: Answered agree even though he was thinking of a job he found very boring. He said he would not have agreed if he’d been thinking about the roofing job he found interesting. For this R, an Agree response meant he often got bored.

#9: R said she had to “switch everything around” because it was phrased as “a negative.”

Work Support

The three questions in this series were more closely related than earlier questions, based on how the responses clustered. Six of the respondents answered all three questions the same (all 1s or all 2s).

In the sequence, if not ALL people at work were supportive, people one could learn from or people one could turn to for help, then several Rs did not answer Strongly Agree. They dampened their agreement if there were some people or department who did not meet these characteristics.

1. People at work are pretty supportive of me

Generally the idea of supportive people at work was well understood. Rs tended to talk about co-workers or direct supervisor caring about how they were doing, whether they were struggling with their job, giving encouragement and positive feedback.

#2: “I think the big thing about being supportive is helping someone when they are in need. We will go back to the roofing job. I will be doing something and someone could see that I was maybe struggling a little bit maybe trying to pull up a piece of plywood and its stuck. I am trying to do it myself because I want to get the job done and people are standing around and eventually somebody looks and notices and says hey you know I’ll give him a hand. That is being supportive adding that extra hand, or it goes as far as to say nice job to somebody. I always try to confirm to people that they are doing a good job and then it comes back in my favor 90-percent of the time.”

#25: “You can’t really rely on positive feedback all the time from the employer. So that’s why I said I neither agree, nor disagree, because sometimes you’ll get credit, sometimes you won’t.” (R was not thinking of peers, just of boss)

#28: “They ask if you need anything if you’re real busy, you know, tells you’re doing a good job, like gives some feedback when you need it.”

2. There are people I can learn from at work

Some Rs were thinking about learning about the job or how to do job tasks, while others were thinking in wider terms and saying things like “everyone has something you can learn from.”

Otherwise, the question worked fine, with the caveat expressed above for this whole sequence.

3. There are people I can turn to for help in solving a work problem

Several respondents stressed that some people were not approachable or willing to help, or just would not be able to help, but others were.

#2: “Yeah, there is always somebody that knows what they’re doing. I always try to figure out the problem myself. If a question comes up or I don’t know to do it, you go and ask someone else. “

#25: “I feel that there are people that you work with that you would be able to go to for advice, or to help resolve an issue, and there are some people who will either escalate the issue, or make it work. So there’s appropriate people that you go to for problems or issues.”

Findings from Testing Form B²

Questions about Your Current Job

Please answer the following questions thinking about your current job. Circle one answer for each question, indicating how strongly you agree or disagree with each statement.

Respondents focused on their current job (or their main current job) without difficulties. They reported being comfortable with the response scale provided, which they have dealt with before. They commented on the fact that job or school applications sometimes include questions in this format.

The jobs they had in mind while answering were:

#5: Artist representation and public relations work

#6: Construction work, building and repairing homes

#7: Financial consultant. “I work with companies, anywhere from small, to mid to large size companies as far as doing their employee benefits, profit sharing plans, things such as that. I’m really just kind of assigned to current accounts, and I maintain relationships, and help build more sales, really, within the companies that I work with”

#10: Makeup artist. “My main gig is seasonal, so it’s sort of October until April. And then I do freelance work in films, and commercial TV, that sort of thing kind of over the summer, and wherever I can squeeze it in.”

#11: Retail sales associate

#12: Manager in a spa

#15: Graphic designer in government agency

#16: Server at a restaurant

#24: Driver and dispatcher at towing company (family business)

Questions about Your Current Job

The intent of this set of questions showed to be ambiguous in administration. The questions could be aimed at finding out what respondents feel that staying over a longer time will let them acquire other people’s respect, eventually get to do satisfying work, eventually make enough money…, or eventually work with other people who share their interests and values. Or, they could be about continuing to get all those things (which, it would be implied, they are already getting at this job). R #7 answered thinking of gaining more respect in the future, while R#16 mixed time frames in her responses: “Getting respect, I answered about current. The others, I interpreted as “Will allow me in the future.” I was thinking like if I stay there forever, how would I feel?” The other respondents answered about their present situation at work, not about the future.

1. Remaining at my current job will allow me to … get respect from other people

Interpretation of ‘getting respect’ ranged from getting promotions, being hired sooner at a new job, “Respect in the sense that I just think of the way in which I am talked to, the way in which I’m interacted with,” “being professional with your co-workers,” prestige for the position in society at large,

Who are ‘other people’? Some respondents interpreted the question as referring to other people at work, while others answered about people in general.

#12: This is the only one that I actually had a pause, and I wasn’t sure if it was other people in my job, other people in the community. I kind of answered it from the perspective of other people in general in the community, my friend circle, my family and things like that

#15: This R was not sure how to answer so he answered 3. He said the question was ‘too vague’ but could not explain why. This R is a project hire. His term of employment will end in December. He knows he won’t be allowed to remain on the job beyond that time, which is bothering him a great deal. Therefore, he answered most of this series as Disagree, because the clause ‘remaining at my current job’ did not make sense in his case.

#24 Answered between agree and neither/nor because it depends on the people, some do and some don’t respect his job.

2. Remaining at my current job will allow me to … do work that I find satisfying

Other than the general note about this series of questions above, this question did not pose any particular problems. It was interpreted across the board as intended.

3. Remaining at my current job will allow me to … earn enough money for the lifestyle I want to have

A few respondents associated ‘lifestyle’ with high lifestyle: nice cars, travel, dining in expensive restaurants. Others more neutrally, as the style of life they want to live. Of interest:

#5: This R answered about the lifestyle, unrelated to the money. She does not make the money but because she is around artists, she lives some of their lifestyle.

#7: This R marked strongly agree when he meant strongly disagree—he thought his mistake was probably due to thinking about the fact that at present he makes more than he needs, but that is not the lifestyle he wants for himself in the long run.

4. Remaining at my current job will allow me to … work with other people who share my interests and values

For one of the respondents (#7) this question was double barreled: he explained that he strongly agrees as far as values are concerned, but that his interests are very different from other people at work.

No other issues identified.

Job Satisfaction

1. I feel fairly well satisfied with my present job.

Only one R, #12, answered Strongly Agree. Three disagreed, and the remaining answered in between. There were no interpretation issues. Level of satisfaction had to do with R expectations, whether the job was in the type of work they want to be doing whether they felt they were successful in their jobs, etc.

2. Most days I am enthusiastic about my work.

Several Rs defined this as getting up in the morning and going to work happily, and being happy to be there once they arrive. “It’s that get-out-of-bed-in-the-morning feel.” Those who spoke about lack of enthusiasm, generally focused on the low pay.

No interpretation issues.

3. I find real enjoyment in my work.

Sources of real enjoyment in their work varied substantially across respondents. While some focused on the kind of work they like to do, others focused on work relations, others on the pleasure of knowing you are doing your job well, and even on just on the fact of having a job in this economy.

No interpretation issues.

Job Persistence Intentions

1. I plan to remain in my current job over the next year

Respondents were able to answer this question without much hesitation. Responses were generally independent of the answer to the subsequent question. That is, even though some Rs think about changing jobs, that did not necessarily go hand in hand with actual plans.

No interpretation issues identified.

2. I don’t usually think about leaving this job

This question appears to have low reliability. As they explained their answer, several respondents expressed conflicting feelings and were unsure of what to answer. It may be worth testing quantitatively for reliability.

No problems identified in interpretation.

3. I feel pretty strongly committed to my current job

The notion of commitment to a job varied substantially across respondents. If this is a wider range of interpretations than intended, it may be advisable to make the wording more specific.

#6: “Commitment as showing up every day and doing my job to the best of my ability.”

#10: R interpreted commitment as passion for the kind of work she does.

#11: “Always following through on the work, showing up on time, following through with procedures, putting on a happy face, even when you’re not, trying to get along with all the people that you work with, just doing what you’re supposed to be doing and doing a good job of it.”

#12: “I feel a sense of ownership in the work that I do, and want to honor the relationship I have with my job.”

#15: R answered disagree, because his position is ending.. However, upon probing he said: “I know speaking for myself, like right now, no one really knows that as soon as the project ends, I’m basically making preparations to find something else. But also, at the same time, since I’m so committed to the job, and responsible with everything, and going to each project 110 percent, I don’t think that anyone can assume that I have other goals, or that I plan to leave.”

#16 “Committed meaning you plan on staying. You are committed to working there; committed meaning that’s what you’re already decided to stay.”

#24: “To be at least 85-90% focused on your job”

Overall Respondents’ Comments about the Questions

Generally respondents said the questions were easy to understand and answer. One R commented: ´I think maybe if there was a sentence that will reiterate it to like kind of stay focused on this job, and don’t bring in past jobs.” It may be harder for those with multiple jobs or short term jobs to keep thinking about the same job across questions.

Findings from Testing Form C³

Two versions of form C were tested. The questions in both versions were identical; however, the order in which each topic was presented varied. Five interviews (#s 1, 8, 19, 23 and 30) were conducted with version 1 in which the order of sections was: Education Self-Efficacy, Education Interests, Education Supports, and Education Persistence Intentions. Version 2 was tested in five interviews as well (#s 3, 13, 21, 26 and 29) with sections presented in the following order: Education Supports, Education Persistence Intentions, Education Interests, and Education Self-Efficacy.

Questions about Your Education Experiences

Please answer the following questions thinking about your current educational program. Circle one answer for each question, indicating how strongly you agree or disagree with each statement.

Respondents were in a variety of different programs, some pursuing degrees (Associate’s, Bachelor’s, and Master’s) and some in certification programs.

Version 1 Cases:

#1: in a 2-year certification program for a physical therapy assistant

#8: in a community college pursuing an AA—No major yet, but perhaps in communication

#19: pursuing a BA in political science

#23: pursuing a Master’s in counseling psychology

#30: pursuing an AA—no major declared but thinking about psychology

Version 2 Cases:

#3: pursuing an LPN degree

#13: studying to be a paramedic

#21: in an MBA program

#26: CPA preparation certificate program

#29: pursuing a Bachelor’s in Psychology

Education Self-Efficacy

In Form C v2, where these questions are presented as the last set, there appeared to be less discrimination by respondents across the first three items on confidence and certainty, which all five answered with the same rating across items (either Agree or Strongly Agree for all 3 items). In Form C v1, there was more variation.

1. I’m confident that I can earn strong grades in my education program

Version 1 Cases

Agree = 3 / Strongly agree = 2

Respondents had no difficulty answering. There were no issues of interpretation. What they considered strong grades ranged from A’s only to include up to Bs. Respondents explained their confidence either based on their motivation to do well or on their prior experience that they can get strong grades.

#1: R cited strong motivation as reason for her level of confidence—Strong grades = As

#8: Grades depend on one’s effort: “If you want a good grade, then you’ll get the good grade, and you’ll work as hard as it takes to get the good grade.” As and Bs

#19: “Basically, just looking at everything that factors into academic success, so quality of professor, availability of outside help with tutors or anything like that, factors that are more dependent on me, so like the amount of free time that I can commit to doing homework, versus study, things like that. And in my case, we do have tutors available at the school. The professor, I think, is doing a satisfactory job. And I have budgeted enough for myself that I’m able to have free time in which to work on this outside of class.” B or better.

#23: “Well, just based on the grades I’ve received in the past year and, you know, grades I received from my life in high school and undergrad, I feel like, you know, I’m capable of getting strong grades.” A, A-, B+.

#30: “I am confident, because for starters, I’ll be paying for everything, so if I don’t put in the time and effort, I’d be wasting my own time and money. I don’t have many distractions. I have the time to focus on my studies, and I have been focusing on my studies. The study skills class I took helped me refocus and get back into getting a proper schedule and a proper routine to complete the program.” Honor roll, A minimum of 3.0 GPA.

Version 2 Cases

Strongly agree = 2 / Agree = 3

#3: R was thinking of As and Bs. “I’m very confident. I study. I try and study a lot. I do know that there are some hard courses coming up, so I’m going to work my best to always get good grades. There are times when there are some things that you just don’t understand and your grades in that class might not be the best but you try the hardest that you can. That is why I put agree instead of strongly agree.”

#13: A’s. If it’s a B, I’m after school, whatever I can do to pull it up, because the paramedic program, that’s pretty competitive. They usually want you to have A’s. B’s, you usually have to get on a long waiting list to get in.

#21: B average or higher

#26: B’s and up

#29: A’s & B’s

2. I’m certain that I can understand the most difficult material presented in my courses

Respondents sometimes answered the questions thinking not about the most difficult material presented but about the most difficult courses. Others focused on the difficulty of the material, but answered about understanding it after using all available outside resources, including tutors in school or support from the instructors.

Version 1 Cases

Strongly agree = 2 / Agree = 2 / Neither-nor = 1

#1: Said agree (and not strongly) because “Probably my neurology class was a lot about the brain and certain scientific aspects that you are not—I was a biology major in my under grad and so I do understand a lot of it but there is certain things that you just might never get completely.”

#8: R was thinking only of the courses he is taking this semester. If he were taking math as he did last semester, he knows there is a lot he would not understand.

#19: R feels that being in the first days of his first semester back in school after a number of years, he is not yet sure how difficult the material can get. Thus, he answered 3.

#23: “I feel like the material that’s presented in the past year has not been—it’s been challenging, but it hasn’t been, you know, incomprehensible. I guess it’s just more a matter of how much time I allow myself to stay with the material, versus not being able to understand it.” R answered Strongly Agree.

#30: R said agree despite some concerns that in a community college he will not have the support necessary and will have to make it on his own.

Version 2 Cases

Strongly agree = 2 / Agree = 3

#13: “I strongly agree, because there’s just so much technology out here. You have the Internet. You have all type of library accesses. It’s always somebody that you can talk to about something you don’t understand.”

#21: R agreed that would be the case if one devotes enough time and accesses available resources

#26: “Yeah, I said I agree. There are very hard concepts in accounting, and I try my best to understand them. But if I were to reach out to maybe the instructor more in his time off, and try to understand them more thoroughly, then I would have put strongly agree. That’s why I put agree right now, because I believe that I go the mile, but I don’t go the extra inch.”

#29: R answered not about difficult material but about a difficult course because students were expected to argue and debate, which was hard for her to do.

3. I’m confident that I can do an excellent job on my education assignments

‘Education assignments’ uniformly interpreted across the board as in class or homework given by instructors. One respondent included readings after answering probes made him think of it. He said if he had included readings in his answer, he would have softened his confidence because he knows he does not do nearly all the readings assigned (#23, a grad student).

Version 1 Cases

Strongly agree = 2 / Agree = 3

#1: Based answer on strong motivation to do well.

#8: Based answer on his willpower.

#19: R’s confidence is based on comparing school with a full-time job. He feels school is less work, so he should be able to do an excellent job.

#23: R answered Agree: “I didn’t put strongly agree because I do feel like if I give myself adequate time with material, or if I had better time management skills, for example, then I would strongly agree.” R discussed doing an excellent job in terms of getting high grades, rather than on the quality of the work.

#30: R answered Agree: “It’s more about the amount of effort I put into it. I know that if I can minimize my distractions, I can achieve pretty much—I feel like I can achieve whatever I want. “

Version 2 Cases

Strongly agree = 2 / Agree = 3

#13: R’s confidence is based on prior experience.

4. I’m confident that I could complete my degree despite financial pressures

A few respondents understood the item asked about one’s ability to focus on school work and complete the degree when financial pressures loomed large in one’s mind. Others interpreted as the sheer ability to pay for school (and therefore be able to stay on the program). For some Rs, to express confidence about something that they felt could be out of their control was a difficult notion.

Version 1 Cases

Strongly agree = 3 / Neither-nor = 1 / Disagree = 1

#1: R interpreted item as referring to whether financial pressures could interfere with one’s ability to focus on the program and complete the degree.

#8: R has faced such problems and was able to get loans and proceed with the program.

#19: R answered neither-nor because of concerns about difficulties to get loans.

#23: R has lots of financial support from his family and does not need to worry about financial pressures.

#30: R strongly agreed: “I have support from family and friends that will help me, you know, so money is not really an object this time around. And I do have a little bit of money saved up, so I feel like I’m in great—it’s going to be up to me if I complete school or not.”

Version 2 Cases

Strongly agree = 1 / Agree = 1 / Neither-nor = 1 / Disagree = 4

#3: R agreed. He felt that he has learned the hard way how to get financial assistance. “That way you can earn your degree just the same and still receive more like financial aid and more grant money where you don’t have to take out as many loans. I can find ways around it. There are always scholarships. I do complete different essays and different things like that so whatever it kind of takes where I don’t have to take out as many loans.”

#13: “That’s one of the confusing questions, because you really never know about tomorrow. And then the financial aid is usually pretty shaky. And then I know I don’t have all the funds for it, so I am dependent on the financial aid. So I’m sure that like if there was some type of law passed, or if they cut back on financial aid that would be something that’s like not something that’s in play for now, but you really can’t say if it did ever happen. So that question was kind of like I didn’t know if they was talking about right now. I didn’t know how to answer that one.” R answered neither-nor.

#21: R disagreed. He had recent experience that made him think that way: “Because I just recently failed my first college course period and that was part of the reason why. I’d be in class, sitting there hungry, and not being able to pay attention, so I wasn’t going. And I wasn’t completing assignments. And that was stress, because I wondered how I’m going to get money and all that kind of stuff. So just recently having an experience with that basically influenced the answer that I just gave, because I just went through that.” This R was thinking of inability to focus due to financial pressures.

#26: R disagreed: Yeah, because my financial obligations and pressures are really weighing heavy at the moment. I mean I want to be in the mindset of being confident that I am going to complete it. You know, if something tragic happens, I’m not going to be able to complete it.”

5. I’m confident that I could complete my degree despite having competing demands for my time (e.g., from work or family)

For some Rs this question was closely related to the previous one, because work demands are closely related to availability of funds for school.

Version 1 Cases

Strongly agree = 2 / Agree = 2 / Disagree = 1

#8: R has to work to be able to afford staying in school, which makes her worry about having enough time for school. She does feel the competing demands of work.

#19: R stopped working to focus on school and feels no competing demands. Hence his confidence.

#23: R does not work and does not have family in the area or making demands from afar.

#30: “I agree, but not strongly, because you can’t really expect the unexpected. There’s always something that happens. You know, I help take care for my mother, who’s disabled. There is a lot of outside circumstances that might stop me, slow me down. I can’t predict the future. If I could, I wouldn’t be going to school.”

Version 2 Cases

Strongly agree = 1 / Agree = 2 / Neither-nor = 1 / Disagree = 1

#13: “It’s pretty closely related, because work, that’s the other side of the money. So like if work becomes a little more demanding, like if they wanted me on full-time, I would have to, I mean the schedule’s pretty flexible, but I really can’t tell about tomorrow. So it’s kind of hard to answer questions about tomorrow when you are not really certain what tomorrow’s going to be like.”

#21: R disagreed: “Pretty much it’s from I’m not confident, not from the work, but from the family standpoint, because like I said, I just tend to get bugged about everything every five-minutes. And pretty much all this revolves around, once again, like I said, being able to manage your resources, being able to manage your time, and discipline, and being able to organize yourself, and focusing and concentrating. All of these are intangible qualities and characteristics that the culture of my family is not—I feel in my humble opinion does not understand. You cannot be pursuing any kind of goal with the attitude of I’ll put it off, and I’ll wait till the last minute. That leads to inefficient assignments, and inefficient productivity. So pretty much that’s why I said I’m not confident, because the culture of my family is not conducive to the qualities that it would take to be successful in education.”

#26: R agreed, but in giving an explanation R contradicted himself: “I guess it depends how heavy the demands are from my family or work. Well I’m out of work, so it’s probably from the family. Like if my mother gets sick and really I wouldn’t be able to complete my degree.”

Education Interests

1. I really like my education program on the whole

What respondents understand by ‘education program’ varies. Some talk about their interest in the career they have chosen, the occupation they are training for, or the courses they are currently taking.

Version 1 Cases

Strongly agree = 1 / Agree = 3 / Disagree = 1

#1: Strongly agreed: “It’s what I want to do now. It’s part of my occupation. You have to like it otherwise you shouldn’t do it. I do like it. It’s very interesting. It’s hands on so you can really-really see the meaning of what you are learning, and that is why I enjoy it.”

#8: R focused on the courses he’s taking this semester.

#19: “Yeah, I’m interested in the subject matter. I think that the faculty and the adjunct professors are pretty interesting, and certainly interested in the topic that they’re teaching. And I think that makes all the difference. When you’re actually interested in something that you’re studying, and when you have somebody teaching it to you that’s equally interested.”

#23: “I don’t know, just maybe like frustrations, and methods in terms of expressing criticism about professors, or classes, and just things on those lines. But I guess at Northwestern, we’re on a quarter system, and I’m not a huge fan of that. I think it’s just too much pressure, and you don’t have enough time with the material… But then again, I chose to go to Northwestern, so I knew what I was getting myself into.”

#30: “Well the program, you know, I don’t feel like it’s customized to—you know, I don’t feel like there is really much, you know, they give the classes, and everything is out there, but it’s pretty much you have to pick your own program. …That’s why I would neither agree, nor disagree with it, because it’s pretty much me. I picked it on my own. Counselors weren’t as much help.”

Version 2 Cases

Strongly agree = 1 / Agree = 3 / Between Agree and Neither-nor = 1

#3: “Like for the medical assistant program I just felt like we have to take a pharmacology program. Within that program medical assistants are not allowed to write prescriptions. We are not even allowed to administer medicine. So, I just didn’t feel there was a need for that program. As far as that, that was probably about it. Now some of the courses were a little bit, were a little hard I guess, AP, Anatomy and Physiology, you know to learn all of the terms and memorize them, and you know how to talk in a certain language, the medical language and everything it can be a little frustrating, but I made it through it pretty good.”

#13: “I agree, because you’re just really hands-on, a lot of practicals. It really gives you a good feeling, so that when you do go out into the field, oh yeah, we went over this is class, or we went over this in the study, or I remember talking to my teacher about this. I’ll run into it, and then I already kind of know what those are signs.”

#21: “The fact that I’m learning something I can use, the fact that it’s a convenient class. I only have to go one day a week. I also like the fact that the faculty are people who are working in the field, as compared to my undergrad, some of them weren’t necessarily in the field, per se. I just feel like you could relate to them more.”

#26: “I said agree. Some of the professors are helpful. Some of them are not helpful, though I would like them to be. The books are okay. It’s a typical institution. You know I mean I probably may lean towards the neither agree, nor disagree, because I don’t think everybody learns the same way, you know, off of just reading a book. I’m a visual, hands-on learner. I would probably have to change that to neither agree nor disagree if that is okay. What do I like? I like some of the people that I bump into in class and the networking. I like learning new concepts. I get along with most of my professors. So that’s what I like about the program.”

2. I am enjoying the courses in my field of study

This question was problematic for those who do not have a field of study, still doing general education requirements for college. For those who already answered the prior question thinking of courses, this question seemed repetitive. No interpretation problems detected.

Version 1 Cases

Strongly agree = 1 / Agree = 3 / Neither-nor = 1

#30: R has no major declared but in answering was thinking only about courses he is interested in, and excluding other required ones.

Version 2 Cases

Strongly agree = 2 / Agree = 2 / Neither-nor = 1

#26: This R was impatient to complete his program and start working in the field. “So I’m semi enjoying what I’m learning, and kind of not at the same time, because I want to use it in real life.”

3. I get totally absorbed in what I am studying

There was variation in the interpretation of “Getting absorbed.” For some Rs this referred to the moment when one is in class or doing school assignments and can concentrate to the point of becoming oblivious to everything else, being very focused on the material at hand. For others, it referred to school work taking over all other aspects of life.

Version 1 Cases

Strongly agree = 1 / Agree = 3 / Neither-nor = 1

#1: “Don’t get distracted by other things going on, really be interested in it when you are actually reading it. Not as reading it and forgetting it. It actually sticks in your mind, and you memorize it kind of something.”

#8: “No distractions while working on assignments”

#19: R answered Neither-nor: “As a blanket statement, I don’t think that that works. I think at times, I’m definitely very interested and absorbed in what I’m working on. I think there is boring parts to everything though.”

#23: “I guess I’m just not the type of person to get really enmeshed in his work. It’s just it hasn’t happened for me.” R did not think of absorbed in a specific task, but more generally.

#30: R used ‘absorb’ both in the intended sense and in the sense of taking in the material: “I feel like I’m able to focus for once, so I can get absorbed. You know, I can drown out, I turn my phone off and put it in the other room, and just focus on what I need to do. I feel like I’m absorbing it. I feel like when I’m in class, I’m a sponge, and I just soak up as much information. I’m not trying to like talk to the girl next to me or anything like that. I’m there to learn.”

Version 2 Cases

Strongly agree = 1 / Agree = 2 / Neither-nor = 2

#3: “that I’m 100-percent all about school. Nothing else matters but school but this program and it’s not like that for me.”

#13: “Absorbed, to really take it all the way in. To be absorbed means to soak it all in like a sponge, just take in every element of it and just embody it.”

#21: “Absorbed means like you’re just oblivious to everything else.”

#26: “Like you’re focused, just you and the material.”

Education Supports

1. I feel support from important people in my life for completing my education program

Respondents spoke both of emotional support and financial support. No interpretation issues. Some Rs had difficulty answering because they had some important people in their lives who give support and others that don’t.

Version 1 Cases

Strongly agree = 5

#8: R feels his full family rooting for him; he will be the first grad in the family.

#19: “They have been very supportive of all of my choices, but particularly to go back to school. So my parents have definitely gone out of their way to help me with this, and my sisters, as well. I mean everybody just seems to be kind of pulling for it.”

#23: “My parents are supporting me financially. You know, they’re always like encouraging me to do that. Maybe not at first, I don’t know if at first they were excited about me becoming a therapist, for various reasons, but in general, they are very supportive.”

Version 2 Cases

Agree = 2 / Neither-nor = 1 / Disagree = 2

#3: R disagreed. She was thinking about her family who does not give her support. She was also thinking about financial support. She gets help and moral support from her boyfriend but was not thinking of him in answering this item. She said she thought about him in subsequent items.

#13: R marked Agree. “For the most part, I didn’t want to mark disagree or strongly disagree, because there is at least one or two people, so I just kind of focused on the one or two people.”

#21: R disagreed: “Yeah, I don’t feel them to be supportive, because to me, support is working with me and understanding. And when you’re egotistical and egocentric, just calling it what you want, instead of creating a mutual understanding, then there’s no support.”

#26: “Because some people support, and some people don’t. And it’s just I want to go with my gut feeling to finish it and whatnot. So that’s why I put neither agree, or disagree.”

2. My family members support my choice of education program

Some Rs indicated that their family does not know much about their education program. Rs know more clearly if their families support their furthering their education, but when it comes to the specific program, not all families are involved, and that makes it hard to answer the question.

Version 1 Cases

Strongly agree = 3 / Agree = 1 / Neither-nor = 1

#1: “They know my interest in it. My dad and my brother are massage therapists so there is kind of a background there so it seems like everybody supports that.”

#8: R did not answer specifically regarding choice of program. He said: “Well I mean they only know what I tell them. I’m completely 110 percent independent, and whatever I decide to do is what I decide to do. They’re just there to back me up.”

#19: “I didn’t really know what to interpret the question as, whether it was referring to like a major, like a political science, or just in being a program at all, so I interpreted it as the latter. They certainly want me to be back and school, and are happy that I’m there.”

#23: “I think now they do, now that they see sort of that there are a lot of job opportunities. I think one of my parents, in particular, was very focused on… they wanted me to get a PhD or something. I think that was more part of if then the actual counseling. It’s better now.”

#30: R answered neither-nor because some family members did and others did not.

Version 2 Cases

Strongly agree = 1 / Agree = 2 / Neither-nor = 2

#3: R answered neither-nor because her family is not involved.

#13: R marked Agree, even though he explained his mom and girlfriend are very supportive in every way while the rest of the family is not.

3. I have access to a good mentor or advisor who can offer me advice and encouragement

Some Rs wondered if this was asking about a mentor or advisor at school or outside of school. Some did not focus on a single person but rather on multiple people. At least in one occasion, the respondent made a distinction between advice and encouragement.

Version 1 Cases

Strongly agree = 2 / Agree = 1 / Neither-nor = 1 / Disagree = 1

#1: R wondered if at school or outside but decided to answer about the program. R would expect advice from school and encouragement from friends or family.

#8: R answered thinking only of his family. Upon probing, he said: “now that we’re talking about it, I’m thinking I formed an outside relationship with my English professor last semester, and I know that in any type of qualm that I might have, she would be the go-to girl.” For R: mentor is outside of school, advisor at school.

#19: “So within the context of that mentor or advisor being like somebody in a professional capacity at school, no. They’re there, but it’s sort of the same scenario that I painted with the financial aid officer there, understaffed and overworked. I ultimately looked at quality and service efforts. The other way that I could have seen it interpreted is just in general in my life, do I have that. And I would say yes, I do, but not knowing what the question is, I chose a neutral answer.”

#23: “Oh, definitely, although I don’t really have like an academic advisor. …I do have other people that have been supportive. In our first year in the program, we had this—it was just like a part of one of our classes is called our back home supervisor, which is like we’d go there and talk about sort of our experience in the counseling room, versus talking about our clients. And that helped us a lot, and we can kind of talk about almost, but like almost everything in terms of counseling is on the floor to discuss. That was very helpful. And then I also have my own therapist, so I could talk about different things about my program there. So I’ve had access to different mentors, for sure.”

#30: R was thinking only about in-school mentor or advisor.

Version 2 Cases

Agree = 3 / Disagree = 2

#3: R sees a mentor as more involved while an advisor advises about specific things.

#21: R agreed but was not thinking of a single person, rather about his circle of friends.

#26: R was only thinking about in school.

Education Persistence Intentions

For this set of three questions, all five v1 respondents gave the same answer for each of the three items (either strongly agree or agree for all 3). In v2, all but one respondent also gave the same answer across the three items. In explaining each response, they referred to very similar notions of determination and motivation.

1. I am totally convinced that I will complete my education program

Respondents did not have problems in answering this question. No interpretation issues were apparent. Most responses focused on student motivation to complete the program.

Version 1 Cases

Strongly agree = 4 / Agree = 1

#8: “Well, because I’m certain that I want to finish school, and so usually, if you want to do something, you do it.”

#19: “I made a realistic timetable of how I’m going to progress through coursework. I started budgeting for this like a year-and-a-half ago, so I have that mostly figured out. You know it’s a lot of the scary variables that you have to plan for. And truly, the only thing that’s left is actually performing well enough in the classes to pass them. So I think as long as you can take those controllable steps and get them under control early enough, it becomes much more realistic proposal.”

#23: “I think so. I mean I’ve done well in my first year. I can’t see myself sort of falling apart in the second year. I think I should be okay in terms of finishing. I don’t think I’ve ever had that happen before, so it would kind of surprise me.”

#30: “I don’t feel like failure’s an option. There is no way that I’m going to go through this a second time and fail.”

Version 2 Cases

Strongly agree = 2 / Agree = 2 / Neither-nor = 3

#3: R did not pick Strongly Agree (chose Agree), because she may decide instead of completing the program to continue studying toward a higher degree (RN instead of LPN). She would then complete the requirements but apply them toward a different program.

#21: Neither-nor because ‘anything could happen’

#26: R is not convinced because of financial worries.

2. I think that earning a degree or certificate in my field of study is a realistic goal for me

Some Rs felt this item overlapped a lot with the previous one. There was consistent interpretation of ‘realistic’ across interviews.

Version 1 Cases

Strongly agree = 4 / Agree = 1

#1: So, to me it’s definitely realistic because I know that I’m doing well and there are no other distractions for me.

#8: “a realistic goal is something that’s real, and not just far-fetched. So a realistic goal would be like okay, I can complete a degree in two-and-a-half to three years. That’s realistic. Okay, I would like to complete my degree by the end of this year, not realistic.”

#19: “I thought that it was the same as the question we just went over, just in other words.”

#23: “I mean it’s very similar to the first question. I think it’s very realistic. I mean graduate school is challenging, but I think I can get through it, and be okay.”

#30: “Exact same thing as above. Failure is not an option this time around.”

Version 2 Cases

Strongly agree = 2 / Agree = 3

#3: “Like is it something that you can achieve realistically within a certain amount of years, within a timeframe and everything. I guess like if I was going to be a heart surgeon, the question would be am I going to stick to that or am I going to take so many years, or am I going to drop out and everything. So, is it realistic? Is it something that I think I can really achieve.”

3. I am fully committed to completing my education program

No interpretation issues. Respondents interpreted ‘full commitment’ consistently.

Version 1 Cases

Strongly agree = 4 / Agree = 1

#1: “That means not letting distractions like work and family, or whatever get in the way. You know putting all of your strength forward and trying to do your best and not just getting by.”

#8: “Yes, and I am fully committed to it. Like I said, I took off last semester because I didn’t really know what I wanted to do. I didn’t want to be wasting my time, the government’s money on just classes that I didn’t know what I was doing. And plus, I just wanted to make sure that I had financial security, so I looked for a job and I found one. And now that I have that, I’m acclimated there. I feel more confident that I can start school knowing that I don’t have to worry about necessarily where my next paycheck is coming from, how I’m going to be able to survive, and get to school and things like that.”

#19: “For me, it’s realizing what the purpose of being there is in the first place. I went to college right out of high school because it just seemed like the next logical thing to do, because I didn’t know what else to do, and for that reason, I’m absolutely certain that’s why I didn’t finish. I got to a point where I just didn’t really know why I was there.”

#23: “I mean with commitment comes sacrifices. In a sense you might sacrifice certain things, like friendships, or like not having as much free time, not making money for school. So having all those things shows signs of commitment to a program. Yeah, I mean I’ve already gone through a year of this, over a year of those things. So I would think I’m committed.”

Version 2 Cases

Strongly agree = 2 / Agree = 3

#3: “Finish up the program whatever it takes to finish the program.”

#13: “Strongly, because nothing’s going to get in my way for this. When I hear committed I think about married to, like I’m just kind of joined to, or bonded to.”

#21: “Committed means that you’re dedicated. Commitment means that you’re willing to do the necessary actions. You’re willing to take the necessary actions, and do the things that have to be done to achieve what you want.”

#26: “I didn’t put Strongly Agree because if a job came into play, I would be more committed to that than the education.”

1 Analysis of Form A interviews was conducted using the interview transcripts that appear in a separate appendix.

2 Analysis of Form B interviews was conducted using the interview transcripts that appear in a separate appendix.

3Analysis of Form C interviews was conducted using the interview transcripts that appear in a separate appendix.

2-1