Appendix D - Cog Interview Results

QUALITATIVE RESEARCH FINDINGS FOR B&B:16/17 COGNITIVE INTERVIEWS: BACCALAUREATE AND BEYOND LONGITUDINAL STUDY

Prepared for: RTI ON BEHALF OF NCES

Prepared by: SHUGOLL RESEARCH 7475 Wisconsin Avenue Suite 200 Bethesda, Maryland 20814 301-656-0310 w ww.shugollresearch.com

January, 2017

TABLE OF CONTENTS

Section		Page
1.0	Background and Research Methodology	3
2.0	Research Objectives	6
3.0	Conclusions and Recommendations	7
4.0	Detailed Findings by Probe	9

Appendices

A-1

1.0 BACKGROUND AND RESEARCH METHODOLOGY

Background And Purpose

• RTI, on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, contracted with Shugoll Research to conduct qualitative in-depth interviews with respondents who graduated with their bachelor’s degree between July 1^st, 2014 and June 30^th, 2015 to obtain feedback on some of the questions that will be used in the Baccalaureate and Beyond Longitudinal Study. Feedback from the research will be used to make adjustments to the full-scale interview.

Usability Methodology

• During the cognitive interviews, usability testing was conducted by allowing respondents to complete the survey either on a computer (PC or laptop) or on a mobile device (smartphone or tablet). This allowed respondents to provide feedback on versions of the questions formatted for different devices.

• Shugoll Research utilized a remote cognitive interviewing/usability methodology where respondents used their own computer or mobile device to complete the NCES survey. Remote testing is convenient/flexible for respondents because they can schedule the session to fit their needs and can participate from their home, work or other location. It allows respondents to use the questionnaire in a real-world environment rather than in a lab setting. It is also slightly more cost efficient than conducting in-person interviews in a research facility.

• Shugoll Research’s web-based remote interviewing/usability solution includes webcam technology, streaming video and an audio connection to provide real-time, face-to-face interaction between the respondent and interviewer/facilitator via a shared desktop on their computer. Tech checks were performed with respondents prior to each interview to minimize technological difficulty during the sessions.

• Approximately half of the respondents used their computer to complete the survey, and approximately half used their mobile device (either a tablet or smartphone). Respondents recruited for usability testing on a mobile device were sent a document camera so that live feed of how the respondent proceeded through the mobile version of the survey was visible to the respondent, facilitator and client observers.

• At the end of each interview, the facilitator conducted a debriefing session to obtain feedback on how easy or difficult it was to use the computer or mobile device to answer the questions, and to ascertain ways the questions could be modified to improve mobile usability.

• Clients were able to log-in and watch all interviews remotely.

Cognitive Interview Methodology

• A total of 30, 90-minute, in-depth, virtual interviews were conducted between October 26^th, 2016 and January 6^th, 2017 with respondents who completed their bachelor’s degree between July 1^st, 2014 and June 30^th, 2015. The interviews were originally scheduled to last 90 minutes, however most took 60 minutes to complete. The 30 interviews were broken out into the following segments:

• 2 interviews with Pre-K teachers

• 5 interviews with K-12 teachers

• 23 interviews with “Other” occupations other than a Pre-K – 12^th grade teacher

• Respondents were recruited by a national consumer database.

• Respondents participating in this study had to meet the following recruiting specifications:

• Must have completed the requirements for their bachelor’s degree between July 1^st, 2014 and June 30^th, 2015. Note: A few exceptions were made for students who qualified but graduated between May 1^st, 2014 and July 31^st, 2015.

• Must have been employed at some point since receiving bachelor’s degree in 2014 or 2015.

• Must be over 18 years of age

• Respondents were recruited to represent a mix of undergraduate institutions and demographics such as gender and race.

Recruiting Disposition

Limitations

• A qualitative research methodology seeks to develop direction rather than quantitatively precise or absolute measures. The limited number of respondents involved in this type of research means the results should be regarded as directional in nature, and they should be used to generate hypotheses for decision making.

• The non-statistical nature of qualitative research means the results cannot be generalized to the population under study with a known level of statistical precision.

2. RESEARCH OBJECTIVES

The specific study objectives were to:

• Obtain reactions to the mobile-optimized version of the survey and determine how easy or difficult it was to use a computer, tablet and mobile phone to answer the questions

• Identify questions that were confusing or ambiguous

• Identify specific terminology that was unfamiliar

• Understand how respondents interpreted some of the questions in their own words

• Understand how respondents determined their answers to some of the questions

2. CONCLUSIONS AND RECOMMENDATIONS

• Many agreed that most of the survey questions were straight forward and easy to complete. Therefore, significant changes to the functionality of the survey were not necessary.

• However, there were some questions that were confusing and need to be revised:

• Many respondents were confused by the “unique jobs” question and didn’t understand what the question was asking. Since their responses to this question determined which subsequent survey questions they received, it is critical that RTI clarify this question so that everyone clearly understands how to respond to it. The terms “unique” and “jobs” were particularly confusing to respondents. As such, RTI might consider alternative terms such as “different,” “separate,” “roles,” “responsibilities,” etc.

• Many respondents experienced issues with the “industry” and “job title” questions and became frustrated that they were forced to pick from a list of options they felt did not really fit them. Ideally, respondents should be allowed to type in their own industry and job title into an open text field that RTI can then code on the back-end in order to get the most accurate responses.

• Respondents who worked in job settings where their hours changed frequently had trouble answering the “average number of hours worked” question series. Because their hours changed so frequently, these were difficult questions to answer and some felt the information seemed “trivial,” particularly having to remember the exact month and year their hours changed. Many respondents were simply guessing how many times their average number of hours changed, what they changed from and to, and when exactly they changed, and were not necessarily providing accurate information.

• Some questions that asked for the month and year were difficult for respondents to answer (e.g., when they got a pay raise, when they started making vehicle payments, etc.). Remembering the specific month seemed like a minute detail some could not quite remember. Consider only asking respondents for the “year” unless RTI is ok with some respondents guessing and providing inaccurate information.

• Because the survey asked the same set of questions for multiple employers and numerous unique jobs within an employer, a few felt the survey would be too long, tedious and redundant. They said they would have stopped taking the survey because it would frustrate them. The current survey was manageable for 2014/15 graduates who have had few employers since graduating, but would likely be too tedious for less recent graduates who have had more employment opportunities.

• Moderator observation: There were many instances where respondents seemed to gloss over some of the instructions or definitions within the survey questions. This happened with respondents of all ages, but seemed to happen even more often with younger respondents who simply were not interested in reading a lot of text. RTI might consider the use of bold or underlined text to emphasize particularly important instructions or definitions within a question.

2. DETAILED FINDINGS BY PROBE

1. BB17DLICFILT (Professional licensure)

• Respondents agreed that it did help having the examples in the question so these should be kept.

• Some respondents were not sure whether other certifications outside of those listed in the examples would qualify.

• Some respondents seemed to be confused between being “certified” and getting a “certification.” They didn’t seem to know the difference between the two terms. RTI may need to provide additional explanation to clarify this difference.

“Yes, I am a licensed teacher. The examples were helpful.” (Teacher, Tablet)

“What does an IT certification really mean? You can take one class on InDesign and this technically counts as being certified. I would need them to clarify what they mean.” (Other, Computer)

“It’s really straightforward and all of the examples did help because there are so many certification options.” (Other, Computer)

2. BB17DNUMEMP (Number of employers)

• Most respondents found this question easy to answer and they were very confident in their responses. Not much time has passed so it was pretty easy to remember how many employers they had in the specified timeframe.

• However, some were not certain whether the word “employers” implies different companies or different supervisors/bosses they’ve worked for within a single company. Some clarification might be necessary.

• In addition, some respondents who had less traditional jobs were unsure whether they should include part-time employment and internships.

“I thought about my employers since 2015 and I’ve had one. It was easy for me because I have only had one.” (Teacher, Phone)

“I have only had one job since I graduated, so I am very confident.” (Other, Computer)

“Does this include all part-time jobs, internships, assistantships and everything? They need to explain this.” (Other, Computer)

“This was a little difficult because I was temping before I got my permanent job so I worked at a lot of different employers at that time. Sometimes I worked somewhere for only two weeks or one month and I wasn’t sure whether to include those.” (Other, Computer)

“I am confused if they want me to count my unpaid internship but I am going to include it. I am confident in the number of employers I’ve had but it is difficult for me to answer this question because I do not know if it is right that I included the unpaid internship.” (Other, Computer)

3. BB17DEMPNAM (Names of employers)

• Most respondents found this question easy to answer.

• One issue with the layout was that the text box seemed to have a restricted number of characters allowed, so a few respondents had to abbreviate the name of their employer.

• A few said it might be helpful if the instructions stated which order they should list their jobs (e.g., start with most recent).

• A few were somewhat confused by the “self-employed” box. If they checked that, do they write their own name in the “employer name” box? Some suggested it might be better to have a different question about whether respondents were self-employed.

“It was easy, but it might be a little easier if it was bigger for me to type.” (Teacher, Phone)

“It was very easy. It is intuitive but the self-employed check box is a little confusing. It should be there instead of a space to write the employer name, I am not sure if having both next to each other make sense.” (Teacher, Tablet)

“It’s easy because I have only had three employers.” (Teacher, Computer)

“It’s a clean and simple interface.” (Other, Tablet)

“Should we only write in the names of actual businesses or can it be an individual person?” (Other, Computer)

“It’s very straight forward to just type in the name. But, I would like the question to say something about what order we should enter them in. I don’t know if I should just make up my own order or if they want the most recent first. It also only gives me the option to check ‘self-employed,’ but what about if it’s part-time or temporary or some sort of seasonal job? Might be good to allow me to check those boxes if relevant. And if I was self-employed am I supposed to just enter my own name for the employer name? They should just ask a separate question about whether or not I’ve been self-employed.” (Other, Computer)

4. BB17DEMPJOBS (Number of “unique” jobs at each employer)

• Many were unsure what “unique jobs” meant – did it mean unique job duties and responsibilities, unique job titles, or interesting jobs people enjoy? A few also interpreted it to mean how many times their job role changed (i.e., the delta), and if it didn’t change at all during the time they worked there they input “0,” which then skipped them out of many of the subsequent employment questions.

• One teacher said that her duties change constantly (e.g., one day I’m a reading teacher, one day I’m a math teacher, etc.).

• Many said they had different job duties every day and didn’t know how to answer the question.

• One or two said their job duties were boring so they had “zero” unique jobs.

• In summary, there was significant confusion with this question and respondents were unsure how to answer, which will lead to inaccurate data. Since responses to this question dictated which employment questions respondents got for the remainder of the survey, it is critical that RTI determine how to clarify this question in order to get accurate responses.

“This is confusing. I am unclear what they are asking for. They should provide an example.” (Teacher, Phone)

“I don’t know what this question means. Is it about my job title? Maybe it’s if you had more job responsibilities, but this is confusing. Unique jobs might mean something different to them than to me, so they should define it in the question.” (Teacher, Computer)

“This is not simple or straight forward. Unique seems like maybe it’s an interesting job, and my job is pretty straightforward. I have a common job and there’s nothing unique about it.” (Other, Tablet)

“I probably can’t provide an accurate number because I have had so many different duties. Also, my current job has different duties every day.” (Other, Tablet)

“This question doesn’t make any sense. Don’t use the word ‘jobs,’ something like ‘duties’ is probably better. Are they asking if you got a promotion? I really don’t know what they’re looking for.” (Other, Computer)

“I wasn’t sure what ‘unique jobs’ means. It’s very confusing. I would refer to it as ‘how many times have you shifted job responsibilities or titles?’ Also, I’ve had lots of shifts in job responsibilities so it was strange to have to quantify this.” (Other, Computer)

“This is very difficult for me. I tutor people and this requires me to do many different things with different families so I am unsure how to define my answer to this question.” (Other, Computer)

5. BB17DJBZIPEX01 (ZIP code of employer)

• Most respondents found this question easy to answer and they were very confident in their responses.

• The layout of the question was straightforward.

• Respondents liked that the city and state were pre-populated when they began typing in a ZIP code. However a few pointed out a few flaws:

• Per the instructions in the question, one respondent who couldn’t remember the ZIP code tried typing in the city in the hopes that ZIP code options would pre-populate, but they didn’t. She felt this should work correctly in order to make it easier for those who cannot remember the ZIP code.

• Another respondent said just the opposite – that if you type in the name of a big city (e.g., New York) and a list of 20 different ZIP codes pops up, there’s no way to know which is the correct ZIP code for their employer.

“I didn’t know the exact ZIP code; I just knew it was in Brooklyn. The question itself is clear but the list that appeared was confusing because I didn’t know the ZIP. I didn’t know what to do so I just chose one that might not be the correct one.” (Teacher, Phone)

“I typed in ‘Washington, DC’ and 25 options pop up and I can’t remember which is the right one.” (Teacher, Computer)

“I am very confident. I like the functionality; it was easy to type in the city name and have the ZIP code list pop up.” (Teacher, Tablet)

“I know it is the right city but I am not sure about the ZIP code. It is clear what they are asking for but I want to have the option to type in the city without a ZIP code.” (Other, Tablet)

“I don’t remember the ZIP code and would have to look it up. I typed in ‘Brooklyn’ but it has a million different ZIP codes so this didn’t really help me. People from big cities might have a tough time answering this.” (Other, Computer)

“It’s very simple and I like that a drop-down with the city names pop up as soon as you start to type.” (Other, Computer)

6. BB17DINDTX01 (Employer industry—predictive coder version)

• Mobile device respondents found this question more difficult to answer from a usability perspective because it required more scrolling.

• Mobile device respondents also had trouble selecting answer choices at the bottom of the dropdown list because the mobile device keyboard popped up and prevented them from clicking on these options.

• Many respondents were surprised that what they considered to be basic industry categories were not listed as options in the drop-down box.

• Some respondents described this question as a guessing game of sorts where they used trial and error to enter different text to see if that would lead to an accurate list of choices in the drop down box.

• Many respondents could not find their exact industry and, therefore, they settled for something that was close, but not entirely accurate. They felt that they should simply be allowed to type in their industry without having to select from a list of options that was not really fitting or accurate.

“I couldn’t find my industry. The drop-down gave me some guidance but it’s confusing for someone like me who didn’t find my industry.” (Other, Tablet)

“The layout makes sense, but I ended up choosing something that didn’t really fit my industry.” (Other, Computer)

“I wasn’t anticipating the drop-down menu so it took more guessing to figure out what words to use to describe my industry.” (Other, Computer)

“It would be easier if you could choose from a list and check a box. I can’t find the option for my industry. They should just give you an option to write in your industry.” (Other, Computer)

“The question is not confusing, but the drop-down menu confuses me. The options that pop up for my industry are not really accurate. Parts of it are, but other parts aren’t and I would want to remove those parts. ‘Education’ is really broad, ‘Financial’ is really broad. ‘Mortgage’ is a subsection of ‘Finance’. I would like to see a drop-down for ‘Finance and Mortgages,’ but they don’t have that.” (Other, Computer)

7. BB17D1IND01 (Employer industry—radio button version)

• This question was easy to answer for those who found their industry in the radio button options provided. However, some thought the radio button options were too broad or general and they would have preferred to select something more specific.

“I like the radio button option because it is easier and more straightforward.” (Teacher, Tablet)

“For me, this second version is easier because my industry is ‘Education’ and there’s no other way to describe it. But, if I worked in a more complicated industry, I would think the first way would be better because it’s more specific.” (Teacher, Computer)

“I liked the text string. For the radio button option I checked ‘healthcare’ but this category is broad and veterinary medicine may or may not be included.” (Other, Tablet)

“This version was easier, but it’s not quite accurate. My industry is ‘Health Education,’ so the radio buttons are too general. The drop-down was more relevant and specific.” (Other, Computer)

8. BB17D2IND01 (Employer industry follow-up item—radio button version)

• Most felt the industry categories listed were too broad or general and they would have preferred to select something more specific or type in their own response.

“The first version was better because I could just type in what I wanted versus searching through a list that may not be correct.” (Other, Computer)

“I like the first version. The radio button option makes me feel like I am doing something wrong since the categories are broad and I am not sure if mine fits in.” (Other, Computer)

“I had to answer ‘something else’ because I think that the ‘Professional, scientific, and technical services’ option is too broad.” (Other, Computer)

9. BB17DNUMPL01 (Employer size)

• All respondents were very confident in their answers.

• However, most agreed that the ranges provided could be improved.

• There’s a huge difference between a company with 6 employees versus 50 employees, so some felt this category should be smaller (e.g., 6-24, 25-50).

• There needs to be more answer categories above 100 employees as there are some companies with thousands of employees (e.g., 100-499, 500-9,999, 10,000 or more).

“I am very confident; I know that the Board of Education branches out into many schools so it has to be more than 100.” (Teacher, Phone)

“If they’re looking for more specific information you might want more categories over 100.” (Other, Tablet)

“Six to 50 is a large gap. I think the lower categories should be 1 to 10, 10 to 25, 25 to 50.” (Other, Computer)

“My answer was an estimate because I’m not really sure. I would change the categories to 10 or less, 10 to 50, 51 to 100, 100 to 500, 500 to 10,000 and 10,000 or more.” (Other, Computer)

“I think the breakdown is good except that 6-50 employees is a significant jump. I would make this range smaller and give more categories.” (Other, Computer)

10. BB17DWKMON01 (Months worked for employer)

• All respondents were very confident in their answers because they haven’t been in the workforce that long.

• However, some felt it was a cumbersome process to have to select each month of their employment.

• A few wondered how to respond if they worked partial months – do they select that month or not?

• A few wondered whether or not to include their internships.

“I am very confident. Nothing about this is confusing.” (Teacher, Phone)

“I am somewhat confident because I didn’t know if I should include my internships.” (Teacher, Computer)

“I’m confident. I like that you can press the buttons and they light up. It works better like this than a drop-down. The buttons were a great touch.” (Teacher, Computer)

“I’m pretty confident. The layout makes sense. But people will round. Like if they start in late September they may or may not include September. They probably need to tell us what to do with partial months.” (Other, Computer)

“I like this format better than scrolling through a list.” (Other, Tablet)

11. BB17D1OCC01 (Occupation coder)

• Many could not find their exact job title and felt like they had to settle for something close, but not exact, which was frustrating. In many cases, the descriptions did not exactly match the job title listed. As a result, many would have preferred the capability to simply type in their own response.

• Some thought there were too many titles to choose from. Others who could not find their job title wished there were more to choose from.

• A few respondents did not realize that their job title would be captured in this question and then piped into subsequent questions. They began typing their title, but when they didn’t see a match from the list of responses, they simply selected “title not listed” and hit “next” before typing their full title. Then, in later questions, their incomplete answers were piped into the question text, which they thought was strange.

• One or two respondents had multiple job titles with their employer and they weren’t sure which job title to enter.

“I didn’t find an exact match and my job is fairly common. But I did like that it gave the definition once you chose an option.” (Teacher, Computer)

“I didn’t know which position within the company they were asking about. They need to clarify.” (Other, Tablet)

“This is really difficult. I would really let people write their job title in.” (Other, Computer)

“It was difficult for me to find. ‘Tutor’ was not there. It gives me a very limited list so I chose ‘job title not listed.’” (Other, Computer)

12. BB17DLICOND01 (Professional licensure required for job)

• For the few respondents who received this question, they clearly understood what the question was asking.

“It’s something that you have to do prior to doing something else. It’s a ‘must’ – had to have the license in order to do this job,” (Other, Tablet)”

“It’s asking if I was certified in anything and I am not, so I answered ‘no.’” (Other, Tablet)

13. BB17EHOWPREP01 (How prepared for teaching)

• Only a few respondents received this probe. However, they agreed that there were options missing from the list because there were so many ways to prepare for becoming a teacher (e.g., degree in education, alternate teaching certificate programs, majoring in education, Teach for America, etc.).

“There are options missing. There are so many routes you can take to prepare for a certificate: a degree in education, an alternate teaching certificate program and majoring in education to name a few.” (Teacher, Tablet)

“Taking the certification tests are missing. You need to pass a test usually in order to get a certification. “(Teacher, Computer)

“If they aren’t going to give you more options they should let you write in something else; like an ‘Other (Specify)’.” (Teacher, Computer)

14. BB17ECRTST01 (State in which certified to teach)

• This question was very easy for respondents to answer.

15. BB17ECRTFLD01 (Subject areas in which certified to teach)

• This question was very easy for respondents to answer because they were able to find their subject areas listed in the answer choices.

“This is very easy to answer and I think that it is structured well.” (Teacher, Tablet)

“It was very easy because they had what I teach on the list as an example – natural sciences.” (Teacher, Computer)

“Easy. This is DOE language. I am used to seeing this.” (Teacher, Computer)

16. BB17EJBTP01 (Type of pre-K-12 teaching position held)

• This question was very easy for respondents to answer and they did not have any suggestions for improvement. The most common answers were provided as options.

• One respondent struggled with “regular classroom teacher” because she was a special education teacher. She felt this answer choice did not accurately describe her role, but none of the others did either.

“It was easy. I do not have any suggestions for improvement.” (Teacher, Tablet)

“This was a little more difficult. I struggle with the word ‘regular’ because I am a special education teacher.” (Teacher, Computer)

“It was very easy; the most common answer (regular classroom) is listed at the top.” (Teacher, Computer)

17. BB17DSTDAT01 (Month/year first began working in job)

• Respondents were confident in the accuracy of their responses since they had not been in the workforce for very long.

“I am 100% confident in my answer; it was not that long ago.” (Other, Computer)

18. BB17DEMPSS01 (Pay rate when first started working at job)

• Most respondents felt this question was easy to answer.

• Some were not 100% confident, but the question stated they could provide their “best guess” so they felt comfortable providing a response they knew was close to being accurate.

• The sample size for this question was relatively small. Findings indicate that this question was much more difficult for people with non-traditional jobs (e.g., sales, contract work, shift work, etc.).

• There were a handful of respondents who did not want to answer questions about their salary.

“I do not want to answer a question about salary.” (Teacher, Phone)

“This is roughly the right answer.” (Other, Tablet)

“It was a little difficult. I think I was paid hourly but I worked on site as a Resident Advisor and they paid for my housing and I didn’t see the breakdown per hour. So I gave you the average amount I took home each month.” (Other, Computer)

“This was very easy. I am very confident.” (Other, Computer)

19. BB17DEMPSC01 (Pay rate changed at job)

• Respondents were very confident in their responses to this question.

20. BB17DEMPSN01 (Number of times pay changed at job)

• Respondents simply counted the number of times their pay had changed.

• A few respondents who worked for companies that did not increase salaries, but did give many bonuses, weren’t quite sure how to answer the question.

• One or two respondents could not remember how many times their pay changed and they said they would have to go back and check their records in order to provide an accurate response.

“I was only there for a year and I know I got a raise once.” (Other, Tablet)

“I counted the number of times I received a pay raise.” (Other, Computer)

“I counted the number of families I tutored because I charge each family something different based on certain factors.” (Other, Computer)

21. BB17DEMPS101 (Month/year of pay rate change at job)

• The layout was straightforward and simple.

• Some agreed it was difficult to identify the exact month that their salary changed.

“I don’t have any comments on the layout, but I don’t remember the date that my new salary began.” (Other, Tablet)

“They layout is very user-friendly. I like the date dropdown because it helps to see the months and years to remind me.” (Other, Computer)

22. BB17DOVAVG01 (Earnings through tips, commission, or bonuses)

• Some respondents who received a one-time bonus thought this question was easy to answer while others weren’t sure whether to include it in their answer or not.

• A few who had unconventional bonus situations or who received multiple bonuses had to estimate their answer.

“We receive annual bonuses. I got one at the end of the year.” (Other, Phone)

“We only received one bonus, which was cash at Christmas.” (Other, Tablet)

“I am unsure of how to answer this. There are aspects of what I did that may be considered ‘bonuses.’ It was my job to take international students to cultural events like the ballet, so I got to do things like that pretty much every month for free. So I will give a monthly average of these activities as my answer.” (Other, Computer)

“I get bonuses based on programs my company runs. The bonuses depend on the performance of the programs, so I estimated.” (Other, Computer)

23. BB17DEMPHS01 (Average hours worked per week when first started working at job)

• Some respondents felt this question was easy.

• Others, however, found it more difficult to answer. Some were not sure whether the question was asking about the hours they clock in at work or the number of hours they actually spent on their work – these are two separate questions. In particular, those with less traditional jobs struggled more because their hours tended to change frequently.

• Some teachers, for example, were unsure whether or not they were supposed to include the hours they spent doing work or helping students outside of regular school hours.

“This was easy because we had to work 40 hours a week.” (Teacher, Computer)

“It was a little difficult to answer. My hours change and I have been working a lot more hours lately. I am not good at keeping track.” (Other, Tablet)

“I had to think about this and then I changed my initial answer. I’m not confident because I don’t remember how many hours I started out working.” (Other, Computer)

“This is difficult because the hours varied so much. When I first started, I was just getting trained. After that I was doing lots of overtime. It was hard to figure out exactly what to put. I’m only 75% confident in my answer.” (Other, Computer)

“It was easy. My hours were four, 10-hour shifts per week.” (Other, Computer)

24. BB17DEMPHC01 (Average hours worked per week changed at job)

• Most respondents felt this question was easy and they were confident in their responses.

• Many were required by their employers to track their hours, so it was easy for them to know whether or not they changed.

“It’s standard for the summer to work 40 hours per week.” (Teacher, Computer)

“I am confident in my answer. We were required to keep track of our hours because we billed our clients by the hour and generally, it didn’t change.” (Other, Tablet)

“I am pretty confident. I started with fewer than 40 full-time hours and then went to full-time once I got trained.” (Other, Computer)

“I worked a lot of overtime so I know I went from 40 hours per week to 45 or 50.” (Other, Computer)

“I am very confident. I know my hours increased towards the end of the internship because we had events.” (Other, Computer)

25. BB17DEMPHN01 (Number of times average hours per week changed at job)

• Several respondents simply guessed and weren’t sure of their answers. Their hours changed so often they didn’t know how to answer the question and found it very confusing.

• Others who had only had a one-time change in hours could answer the question easily.

“This question is really confusing. I mean, who knows? I’m not sure. It changed a lot so I don’t remember and I don’t know what they’re asking.” (Other, Computer)

“I didn’t know what to put. This is a very confusing question. My hours fluctuate all the time and I don’t keep track of them weekly. I have no idea how to answer this. At a job with overtime, it’s very confusing.” (Other, Computer)

“This is difficult to answer because I tutor schedule-permitting. I tried to remember the number of times it would have changed but really I guessed. I think it is about right.” (Other, Computer)

26. BB17DEMPH101 (Month/year when average hours per week changed at job)

• Those with less traditional jobs where the hours changed all the time had a hard time answering this question.

• In addition, a few said it was very difficult to remember the exact month that their hours changed. They felt this was a very minute detail and they were unsure why this level of specificity was necessary.

• Some felt selecting dates from a calendar view might be easier than the dropdown.

“I think it is pretty straightforward. It could help to see the dates in a calendar format; it is what I am used to when answering these types of questions.” (Teacher, Tablet)

“I really don’t remember. This is confusing and I can’t answer it. I definitely don’t remember the month. This is asking you to recall very specific details that people might not remember so people will guess and it won’t be accurate. This is a very confusing section and I didn’t understand what they’re asking. You’re asking me to recall very minute details. How am I supposed to remember how many hours I worked in a particular month and year? Are you asking me to average the hours over my entire time at that job, per week, at first? What’s the timeframe?” (Other, Computer)

“I know my hours changed, but I don’t know exactly when it happened.” (Other, Computer)

27. BB17DLKWRK (Months looking for work)

• Most said this question was easy to answer and they were confident in their responses.

• The layout and functionality of the question was straightforward and easier than the earlier question regarding “months employed” because there were fewer months they had to select in this case.

• Most respondents thought about the months immediately after graduation when they were looking for their first employment opportunity.

“It was very easy and I am very confident in my response. I thought about when I was looking for a job immediately after school and more recently I have been looking for another job.” (Teacher, Tablet)

“I had to think about my job progression. I’m pretty confident. I gave myself two months for job searching so I knew at the beginning of each job that I had given myself two months to find it.” (Teacher, Computer)

“It was easy. I like the monthly selection for this question. It allows you to select months separately which I think makes more sense for this question. I am very confident in my response; I looked for and received a job in one month.” (Other, Tablet)

“It was easy and I am confident in my answer. I thought about the times I was submitting resumes, going to interviews, looking at job postings and talking to people about job opportunities.” (Other, Computer)

28. BB17DIMPBEN (Debrief on employment section)

• Most said the employment section was easy because they hadn’t been employed very long.

• Many respondents only had one or two employers since graduating so they were ok providing this level of detail. However, they anticipated that someone who had been out of school longer and had many more employers would get frustrated providing this level of detail for each employer.

• Most agreed that the most difficult information they were asked to provide were the average number of hours worked per week and specific dates of when their pay or hours changed. Some said it was the number of jobs they had within their employer.

• A few respondents mentioned wanting to know more about what was going to be done with this information. Their concern was that the survey tied their responses to specific employers. They might not answer certain questions honestly (e.g., questions about satisfaction with their job/employer or whether employers offer opportunities for advancement, etc.) knowing that their responses were linked directly to a specific employer.

“It was fairly easy. The most confusing was the number of jobs I’ve had within my employer, but overall it wasn’t too long or crazy.” (Teacher, Tablet)

“It was fairly easy but I was uncomfortable providing some of the information, especially with my current job. I am wondering how you are going to use it and who will have access to it.” (Teacher, Computer)

“It was pretty easy. The dates of changes were the most difficult thing to provide.” (Other, Tablet)

“I’ve only had two jobs so I didn’t mind. If I had 10 jobs, I wouldn’t want to go through all the trouble.” (Other, Computer)

“It’s kind of annoying to provide all of this detail for temp jobs I only worked for a couple of days. I don’t mind doing it for more stable jobs.” (Other, Computer)

“I want to know what’s going to be done with this information. The survey ties me to the employers I listed and some people won’t be comfortable with this. Some people won’t want to honestly answer the salary question or if you’re asking about satisfaction and they’re currently working there they may not want to answer honestly.” (Other, Computer)

“Some questions were more difficult than others like the change in hours and unique jobs questions. The ones that were very specific were harder.” (Other, Computer)

29. BB17ETHNKINFL (Influences on becoming a teacher)

• Teachers did not have any issues with the layout of this question.

• However, some said the scale wording was strange and suggested potential revisions including:

• Should be a “negative aspect” or “factor” not a “negative influence”

• There should be more scale points than just three to capture more nuance in people’s responses

• Include some “personal” influences such as “have teachers in my family so I had good mentors”

“I like the layout of the question. I like that there are three levels to choose from for each choice, I wouldn’t change anything.” (Teacher, Phone)

“It looks fine to me, the only thing that I might change is moving ‘no influence’ all the way to the left, but that might be a personal preference.” (Teacher, Tablet)

“I wish that instead of the three columns it gave more of a scale from 1 to 10. This would be more reflective of my feelings than just picking one of those three options. I also think they need to have some motivating factors too, like ‘there are teachers in my family so I had good mentors.’ They need some more personal factors in there.” (Teacher, Other)

30. BB17EFTCHHLP (Help received from school in first teaching job)

• Respondents interpreted “first teaching job” to mean their first full-time teaching position with their own classroom (not as a teacher’s assistant).

• Some felt that the issues were adequately represented. Others felt the following items were missing: classroom management strategies, strong classroom culture, working with colleagues, assessments, technology in the classroom.

“I interpreted my first teaching job as my first full-time teaching position with my own classroom rather than when I was a teaching assistant.” (Teacher, Tablet)

“I’m in my first teaching job in my first year of teaching.” (Teacher, Computer)

“There should be a lot more issues listed: classroom management strategies, strong classroom culture, working with colleagues, assessments, and technology in the classroom.” (Teacher, Computer)

31. BB17EMOVE (Debrief on Teaching section)

• The questions in the teaching section were not overly difficult, but teacher respondents felt some of the wording was ambiguous.

• Parts of the teaching experience respondents felt were not well represented included: classroom management, dealing with special education or non-traditional students, implementing technology in the classroom, assessments, classroom culture.

“None of the questions were difficult but there was nothing about classroom management or dealing with special education or non-traditional students.” (Teacher, Tablet)

“The information was not hard to provide but some of the wording was ambiguous.” (Teacher, Computer)

32. BB17FLCAREER (Knowing a second language increased career possibilities)

• Respondents interpreted the phrase to mean that knowing a second language opened the door to more job opportunities.

• Many said Spanish was the language most likely to increase career possibilities whereas other languages often did not.

“Spanish may be helpful if you have a need in your career. To be bilingual may help to increase career possibilities.” (Teacher, Computer)

“I think it means it has opened you up to more job opportunities.” (Other, Computer)

“Did knowing Spanish help me get a job? Did that qualification help me stand out from the pack?” (Other, Computer)

“Knowing Spanish might increase your career possibilities; a widely spoken language may help you get a job with an international company.” (Other, Computer)

33. BB17FHOMDAT (Month/year began paying current rent/mortgage)

• Most respondents who received this question thought it was very easy to answer.

• A few said it was somewhat difficult to remember the exact month they began paying their mortgage.

“I had to think about it. I wasn’t quite sure of the exact month. It was a little difficult.” (Teacher, Computer)

“It was very recent so it is easy to remember and the question was straightforward.” (Other, Tablet)

34. BB17FHOMPRV (Any previous different rent/mortgage amount)

• Most respondents who received this question thought it was very easy to answer.

• Respondents knew whether they had their own apartment or home and were paying rent or a mortgage.

• One respondent completely missed the instructions in parentheses and answered “yes” despite the fact that someone else made the mortgage payments on her behalf. RTI might consider use of bold or underlined text for important instructions in the survey questions.

“I was living with my parents at that time and hadn’t moved to my own place yet, so this was easy.” (Other, Tablet)

“This is easy. I had an internship at GW that included housing and then I moved back home to my parents’ house for a bit.” (Teacher, Computer)

35. BB17FHOMEMOS (Months paid previous rent/mortgage amount)

• Those respondents who received this question found it very easy to answer.

36. BB17FHOMAMT (Previous rent/mortgage amount)

• Most respondents who received this question thought it was very easy to answer.

• For those who had lived in multiple apartments, it was slightly more difficult to remember the exact rent amount for the apartment they were living in during that timeframe.

“It was somewhat difficult. I’ve lived in three different places with three different rents. I estimated the highest and lowest and picked a number in between.” (Teacher, Computer)

“It was easy; but I think it might fade in my memory after a time.” (Other, Computer)

37. BB17FCARDAT (Month/year began paying current vehicle payment)

• Some said this was easy to answer because they purchased a car very recently.

• Others said it took more recall to think back to when they made their car purchase. Knowing the exact month was what made this question moderately difficult for some respondents.

“This was fairly easy since I purchased it recently. If it had been years ago it may have been more difficult.” (Other, Tablet)

“This took a little bit of recall. I had to think back to when I purchased my car, then I thought about the year of my car because I purchased it new.” (Other, Computer)

38. BB17FCARPRV (Any previous different vehicle payment amount

• Most respondents who received this question thought it was very easy to answer.

39. BB17FDONATE (Financial contributions to NPSAS institution)

• Some respondents said this question was easy to answer, but others found the wording to be a bit confusing.

• After thinking it through, many interpreted “financial contributions” to mean financial gifts or alumni donations. But, several respondents initially wondered whether they should consider tuition in their response, but then they realized the instructions in the question told them not to.

• Several suggested clarifying the question by providing examples of financial contributions respondents should include (e.g., gifts or donations) versus what they should exclude (e.g., tuition).

“I interpreted contributions to mean that I’ve answered the alumni solicitations that I receive for various amounts of money. I think providing an example of what to include would help just so that I am sure about what they are asking here.” (Teacher, Computer)

“Does this include loans? I’m not sure. It’s worded a little strangely. It probably mostly means donating to the university, but it wasn’t too clear.” (Teacher, Other)

“Does it mean donations to the school? I initially would have thought it meant tuition, but then I read the parentheses and assumed it means donations.” (Other, Tablet)

“I wasn’t quite sure what they meant. Do they mean did I donate? Alumni donations? I had to think about what financial contributions means. They should give examples of what this means (e.g., donations) instead of what it doesn’t mean.” (Other, Computer)

“At first I thought tuition was included, but then I saw their caveat.” (Other, Computer)

40. BB17FAFFCOST (Financial costs of education)

• Most respondents were comfortable answering this question, although they admitted it was a bit personal.

• Some felt sad or “bummed” thinking about this question and how much of their lives they had put “on hold.”

• Most agreed the question was straight-forward and felt it was asking how the burden of student loans had affected any of the life milestones listed.

“The question was o.k.; it does not make me feel good or bad. It is asking if I have been stifled by student loans or other financial burdens related to my education.” (Teacher, Tablet)

“I’m bummed that this is what my deal is. It’s asking how has having student loans and other education debts impacted my capacity to being a fully functioning member of society. Or how the burden of student loans has affected people.” (Other, Computer)

“I don’t mind answering it, but it doesn’t make me happy to think about the financial burden. It’s asking whether as a result of the financial burden, did it stop me from doing these things?” (Other, Computer)

“It made me consider, but I didn’t necessarily feel uncomfortable. But, these things are difficult to think about, it’s very personal.” (Other, Computer)

“It is a good question to ask. I think a lot of people in my generation would answer this the same way, choosing many of these things on the list. The list makes it a jarring reality for me.” (Other, Computer)

APPENDIX: RESPONDENT PROFILE

APPENDIX: RECRUITMENT SCREENER

Eligibility Screening Questions

The recruitment screener, a 3-5 minute phone script, will be used to determine if individuals qualify for study participation in order to test the items identified for cognitive and usability testing for the B&B: 16/17 survey. Recruiters will reach out to respondents or potential respondents will call a toll-free number mentioned in an ad and an experienced recruiter will screen the caller for eligibility for the project.

Intro: Thank you for inquiring about this study. I need to ask you a few questions to determine if you are eligible to participate. The screening is expected to take approximately 3-5 minutes.

Hello, this is . I’m calling from Shugoll Research, an independent market research company, on behalf of the National Center for Education Statistics which is part of the U.S. Department of Education. We are conducting a very brief survey among college graduates and would greatly value your opinions. This is strictly market research; there is absolutely no sales effort involved. I’d like to ask you a few questions to determine if you are eligible to participate. The screening is expected to take approximately 3-5 minutes.

1. Were you enrolled for a bachelor’s degree at a postsecondary institution at any time between July 1, 2014 and June 30, 2015? (DO NOT READ)

	CIRCLE ONE
Yes	1	(CONTINUE)
No	2	(THANK & TERMINATE)
Don’t know/not sure	3

2. When did you complete the requirements for your bachelor’s degree?

__________________________________ (RECORD MONTH AND YEAR)

	CIRCLE ONE
Between July 1, 2014 and June 30, 2015	1	(CONTINUE)
Some other time	2	(THANK & TERMINATE)

3. What is the name of the college or university where you received your bachelor’s degree?

__________________________________ (RECORD NAME OF SCHOOL, RECRUIT A MIX)

4. Have you ever been employed since completing your bachelor’s degree requirements? (DO NOT READ)

	CIRCLE ONE
Yes	1	(CONTINUE)
No	2	(THANK & TERMINATE)
Don’t know	3

5. Since receiving your bachelor’s degree, are you currently or have you ever worked: (READ LIST)

	CIRCLE ONE
As a pre-kindergarten teacher	1	(RECRUIT 5)
As a kindergarten – 12th grade (K-12) teacher	2	(RECRUIT 5)
In some other occupation (specify:____________________)	3	(RECRUIT 20)
Don’t know	4	(THANK & TERMINATE)

6. (ASK IF Q5 CODE 2, K-12 TEACHER) What grade(s) do/did you teach?

__________________________________ (RECORD, ATTEMPT TO RECRUIT A MIX)

7 What is your age? Are you… (READ LIST)

		CIRCLE ONE
	Under 18	1	 (THANK & TERMIANTE)(THANK & TERMINATE)
	18-24	2
	25-29	3
	30-34	4
	35-39	5
OR	40 or older	6
(DO NOT READ)	Refused	7	 (THANK & TERMIANTE)

8. Are you of either Hispanic or Latino origin? (DO NOT READ LIST)

		CIRCLE ONE
	Yes	1	(RECRUIT A MIX)(RECRUIT A MIX) (RECRUIT A MIX)
	No	2
	Don’t know/not sure	3	
	Refused	4

9. What is your race? Choose one or more. (READ LIST)

		CIRCLE ALL THAT APPLY
	White	1	(RECRUIT A MIX)(RECRUIT A MIX)
	Black or African American	2
	Asian	3
	American Indian or Alaska Native	4
OR	Native Hawaiian or Other Pacific Islander	5
(DO NOT READ)	Refused	6

10. Now that you have been out of school for a few years, how has your education impacted your career? (WRITE VERBATIM. PROBE AND CLARIFY)

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT RESPONDENTS IN THIS STUDY BE ABLE TO PROVIDE AND CLEARLY EXPRESS INFORMATION ABOUT THEIR EXPERIENCE. IF RESPONDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE.

• RESPONDENTS WITHOUT A VERBATIM ANSWER DO NOT QUALIFY.

• IF RESPONDENT IS UNABLE OR UNWILLING TO GIVE AN ANSWER IN ENGLISH, THANK AND TERMINATE.

• IF RESPONDENT HAS HEAVY ACCENT OR CANNOT MAKE HIM OR HERSELF CLEARLY UNDERSTOOD IN ENGLISH, THANK AND TERMINATE.

• IF RESPONDENT ONLY GIVES ONE OR TWO WORD ANSWERS AND IS UNWILLING OR UNABLE TO ELABORATE ON MEANING, THANK AND TERMINATE.

11. On a scale of “1” to “10,” where a score of “10” means that the statement describes you extremely well and “1” means that the statement does not describe you at all, how well does the following statement describe you? (CIRCLE ONE NUMBER PER ROW)

	CIRCLE ONE
	EXTREMELY WELL							NOT AT ALL WELL
I feel comfortable expressing my opinion in a group or interview	10	9	8	7	6	5	4	3	2	1

NOTE TO INTERVIEWER: 7, 8, 9 OR 10 MUST BE CIRCLED, OTHERWISE THANK AND TERMINATE

12. RECORD GENDER (DO NOT ASK)

	CIRCLE
Male	1	(RECRUIT A MIX)
Female	2

INVITATION

We are conducting online video interviews with respondents like you to evaluate some of the questions that will be used in an education survey with special topics. This is not a sales meeting of any kind.

The interview will take about one and a half hours to complete and respondents will receive $40 after the interview as a thank you for their participation. The interview will be conducted using online video technology and you can participate from your computer.

A. Can I schedule an interview at your convenience?

	CIRCLE ONE
Yes	1	(ASSIGN TIME SLOT AND CONTINUE)
No	2	(THANK & TERMINATE)

B. Does the computer at the location where you will be participating in the interview have high speed internet?

	CIRCLE ONE
Yes	1	(CONTINUE)
No	2	(THANK & TERMINATE)

C. The computer at the location where you will be participating in the interview must be indoors in a quiet location without any noise that would be disruptive to the discussion. For example, we can’t have loud machinery, dogs barking or children crying in the background. Are you able to find a location to meet these requirements?

	CIRCLE ONE
Yes	1	(CONTINUE)
No	2	(THANK & TERMINATE)
Don’t know	3

D. Does the computer at the location where you will be participating in the interview have a webcam?

	CIRCLE ONE
Yes	1	(SKIP TO QF)
No	2	(CONTINUE)

E. If you don’t have a webcam on this computer, we will send you one. Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

__________________________________________________ (RECORD MAILING ADDRESS)

F. Part of the interview will involve filling out a questionnaire using a smart phone or tablet. Do you have a smart phone or tablet that you would be willing to use for this purpose? (ABOUT HALF OF RESPONDENTS WILL COMPLETE THE SURVEY USING A MOBILE DEVICE)

	CIRCLE ONE
Yes, has a smart phone	1	(RECRUIT 7-8 FOR MOBILE USABILITY AND CONTINUE)
Yes, has a tablet	2	(RECRUIT 7-8 FOR MOBILE USABILITY AND CONTINUE)
No, I have neither that can be used	3	(RECRUIT 15 FOR COMPUTER USABILITY AND CONTINUE)

G. (ASK IF QF CODES 1 OR 2) In order to complete the survey using a smart phone or tablet, we will need to send you a document camera to use to record the screen on your device as you are using it. Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

__________________________________________________(RECORD MAILING ADDRESS)

If for any reason your plans change and you can't keep your appointment, please call and let me know in advance. My telephone number is (INSERT NUMBER).

Finally, we will be sending you a “consent to participate in research form” via email that we will need you to sign and scan back to us in order to confirm your participation.

INTERVIEWER: RECORD APPOINTMENT DATE AND TIME ON FRONT OF SCREENER. COMPLETE NAME AND CONTACT INFORMATION.

Obtain the phone number and email address of the respondent and confirm that the phone number is the best one to reach him/her when they participate in the interview. (RECORD BELOW.)

_______________________________________ (RECORD E-MAIL ADDRESS)

_______________________________________ (RECORD TELEPHONE NUMBER)

THANK YOU FOR YOUR PARTICIPATION.

WE WILL SEND YOU LOGIN INSTRUCTIONS FOR THE SESSION AND ONE OF OUR TECHNICAL PARTNERS FROM INTERACTIVE VIDEO PRODUCTIONS (IVP) WILL CONTACT YOU BY EMAIL OR PHONE TO SET UP AN APPOINTMENT BEFORE THE SESSION TO ENSURE YOUR WEBCAM, DOCUMENT CAMERA, AND THE ONLINE VIDEO SOFTWARE ARE WORKING PROPERLY FOR YOU.

APPENDIX: RECEIPT OF PAYMENT*

*Receipt of payment is included as a separate PDF document

APPENDIX: CONSENT FORMS*

*All signed consent forms are included in a separate PDF document.