Appendix D Cognitive Testing Report

2008/18 BACCALAUREATE AND BEYOND (B&B:08/18) FULL-SCALE

OMB # 1850-0729 v. 13

Appendix D
Cognitive Testing Report

Submitted by

National Center for Education Statistics

U.S. Department of Education

February 2018

QUALITATIVE RESEARCH FINDINGS FOR B&B:08/18 COGNITIVE INTERVIEWS: BACCALAUREATE AND BEYOND LONGITUDINAL STUDY

Prepared for: RTI ON BEHALF OF NCES

Prepared by: SHUGOLL RESEARCH 7475 Wisconsin Avenue Suite 200 Bethesda, Maryland 20814 301-656-0310 www.shugollresearch.com

December 2017

TABLE OF CONTENTS

Section		Page
1.0	Background and Research Methodology	D-4
2.0	Research Objectives	D-6
3.0	Executive Summary	D-6
4.0	Detailed Findings by Probe	D-8
Appendixes		D-32

1.0 BACKGROUND AND RESEARCH METHODOLOGY

Background and Purpose

• RTI, on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, contracted with Shugoll Research to conduct qualitative in-depth interviews with respondents who graduated with their bachelor’s degree between July 1st, 2007 and June 30th, 2008 to obtain feedback on some of the questions that will be used in the Baccalaureate and Beyond Longitudinal Study. Feedback from the research will be used to modify the full-scale instrument.

Usability Methodology

• During the cognitive interviews, usability testing was conducted by allowing respondents to complete the survey either on a computer (PC or laptop) or on a mobile device (smartphone or tablet). This allowed respondents to provide feedback on versions of the questions formatted for different devices.

• Shugoll Research utilized a remote cognitive interviewing/usability methodology where respondents used their own computer or mobile device to complete the NCES survey. Remote testing is convenient for respondents because they can schedule the session to fit their needs and can participate from their home, work or other location. It allows respondents to use the questionnaire in a real-world environment rather than in a lab setting. It is also slightly more cost efficient than conducting in-person interviews in a research facility.

• Shugoll Research’s web-based remote interviewing/usability solution includes webcam technology, streaming video and an audio connection to provide real-time, face-to-face interaction between the respondent and interviewer/facilitator via a shared desktop on their computer. Technology checks were performed with respondents prior to each interview to minimize technological difficulty during the sessions.

• Approximately two-thirds of the respondents used their computer to complete the survey, and the remainder used their mobile device (either a tablet or smartphone). Respondents recruited for usability testing on a mobile device were sent a document camera so that live feed of how the respondent proceeded through the mobile version of the survey were visible to the respondent, facilitator and client observers.

• At the end of each interview, the facilitator conducted a debrief to obtain feedback on how easy or difficult it was to use the computer or mobile device to answer the questions, and to ascertain ways the questions could be modified to improve mobile usability.

• Clients could log-in and watch all interviews remotely.

Cognitive Interview Methodology

• A total of 28, 90-minute, in-depth, virtual interviews were conducted between October 2nd, 2017 and December 5th, 2017 with respondents who completed their bachelor’s degree between July 1st, 2007 and July 30th, 2008. The 28 interviews were broken out into the following segments:

• 14 interviews with respondents who have worked as a K-12 teacher since 2012

• 11 interviews with respondents who have a bachelor’s degree only

• 3 interviews with respondents who have or are pursuing an additional degree or certificate

• Respondents were recruited by a national consumer database.

• Respondents participating in this study had to meet the following recruiting specifications:

• Must have completed the requirements for their bachelor’s degree between July 1st, 2007 and June 30th, 2008.

• Must have been employed at some point since July 2012.

• Must be over 18 years of age.

• Respondents were recruited to represent a mix of undergraduate institutions and demographics.

Recruiting Disposition

Cognitive Interviews 2017
Total number of people contacted (those screened and qualified AND those who did NOT qualify)	1,870
Number of eligible participants screened (how many participants were eligible after completing the screener)	175
Number of people recruited	55
Number of participant initiated cancellations	17
Number of no shows (no participant call to cancel)	9
Final number of interviews conducted	28
The expected no-show/cancellation rate for telephone interviews is 15%. Due to the high commitment of these interviews and the low incentive ($40), the no-show cancellation rate for this study is 47%.

Limitations

• A qualitative research methodology seeks to develop direction rather than quantitatively precise or absolute measures. The limited number of respondents involved in this type of research means the results should be regarded as directional in nature, and they should be used to generate hypotheses for decision making.

• The non-statistical nature of qualitative research means the results cannot be generalized to the population under study with a known level of statistical precision.

2.0 RESEARCH OBJECTIVES

The specific study objectives are to:

• Identify questions that are confusing or ambiguous

• Identify specific terminology that is unfamiliar

• Understand how respondents interpret some of the questions in their own words

• Understand how respondents determine their answers to some of the questions

• Obtain reactions to the mobile-optimized version of the survey and determine how easy or difficult it is to use a computer, tablet and mobile phone to answer the questions

3.0 EXECUTIVE SUMMARY

• Most participants say the survey questions are not difficult to answer, but some say the overall length of the survey is too long.

“It was a long survey with long questions. I would have exited out and done other stuff.” (Teacher, Computer)

• Most agree that the format of the survey is user-friendly, even for mobile participants.

“I liked the layout. It was simple and clear. The hyperlink references were helpful and you don’t have to click on them if you don’t want to. I liked that there was only one question per screen. I also liked the percent completion bar. The use of parentheses was also helpful.” (Teacher, Computer)

“The questions were short enough to find in the window and you don’t have to scroll a lot, which makes it easy.” (Teacher, Tablet)

• One respondent believes it would be easier to complete the survey if there was an ability to skip questions he/she does not know the answer to and then come back to them later after looking up the answer.

“There were some questions I didn’t know the answer to. Can I look up the answer and then go back to it? Can I skip questions and come back later? That would be helpful.” (Bachelor’s degree, Computer)

“If I were taking this in real life, would I be able to pause and come back to it? Would it time me out?” (Bachelor’s degree, Computer)

• Some would like to know more about the purpose of the survey and how their information will be used. One respondent wanted to know whether her answers and her identity will remain anonymous and confidential. She says she would likely change her answers to certain questions if she knew her survey responses were identifiable as opposed to anonymous.

“I really wonder what they’re trying to figure out, especially when it comes to the questions about my spouse.” (Teacher, Computer)

“It gets really personal. Some people might not want to answer those questions because they feel a bit invasive.” (Teacher, Computer)

“They really should talk about what the survey is for and how the information will be used. What’s the purpose? Who’s the survey coming from? What type of confidentiality policy is there? You will get different answers from people depending on confidentiality. They ask for a lot of personal information so I want to know why they need all of that.” (Bachelor’s degree, Computer)

• One teacher made an interesting comment that some of the education questions seem “emotionally charged” (e.g., why they left a school, what they were dissatisfied with, how prepared they felt they were, evaluation of their principal, etc.), and he feels that some teachers could find it difficult to answer these types of questions objectively.

“The type of person who teaches could have a hard time separating their emotions from the job and a lot of these questions are emotionally charged for teachers. It could be hard to be objective.” (Teacher, Computer)

• From question to question, it can be confusing to keep track of whether a specific question is asking about a current employer, previous employer, which job at an employer (if there were more than one) or all jobs/employment experiences in general. Each question might need a qualification such as “thinking about your current employer,” “thinking about your previous employer,” “thinking about all of your employment experiences since 2012,” etc. to get accurate responses.

“Does this question refer to any job I’ve ever had or just my education jobs?” (Teacher, Computer)

“This question is confusing. Are they talking about my must current workplace or something else?” (Teacher, Computer)

• A few participants feel that the gender and sexual preference questions do not seem to pertain to the overall subject matter of the survey – employment and education. However, others were happy to see them included in the survey.

“I don’t understand why the sexuality questions were asked if this survey is about education.” (Additional degree, Computer)

“I don't mind [answering these questions]. I think it's important for minorities to be able to express their experiences.” (Bachelor’s degree, Computer)

4.0 DETAILED FINDINGS BY PROBE

1. RESUME PROBES

Is Résumé Comprehensive?

• Most respondents say their résumé is comprehensive.

• Few others say their résumé is not comprehensive for one of the following reasons:

• They tailor their résumé based on the company/industry to which they are applying (i.e., they don’t include previous positions that are not relevant).

• They are not currently looking for employment so they have not updated their résumé to include their current employer.

“It doesn’t include tutoring, coaching or the random job here and there like summer jobs or catering jobs.” (Teacher, Tablet)

“It has everything except my current job because I am not looking for another job, so until I do, I won’t update it to include this employer.” (Bachelor’s degree, Tablet)

Do dates contain months and years?

• Most say the dates on their résumé contain both month and year.

• A few, however, include only the year.

“I chose to use months and years because it helps to provide a clearer picture of when I was working and address the gaps in my employment.” (Bachelor’s degree, Mobile)

“My résumé has months and years because that is the USAJOBS Format.” (Additional degree, Computer)

Confidence in accuracy of employment time frames?

• Most are confident in the accuracy of the employment time frames on their résumé.

More than one version of résumé?

• Several say they have only one version of their résumé, while most say they have multiple versions since they tailor their résumé based on the company/industry to which they are applying.

“I have different versions for different skill sets. I chose this version for the interview because it is the clearest picture of my employment.” (Teacher, Computer)

“I only have one version of my résumé.” (Bachelor’s degree, Tablet)

“This résumé is the most recent version for the position I just took.” (Additional degree, Computer)

2. B18DNUMEMP

• Participants can either recall from memory or verify on their résumé the number of employers or jobs they have had since July 2012.

• Most are very confident in their answers. One respondent was a little unsure how to respond since she has had multiple positions within one company. She ultimately assumed she should enter just “1” employer, but she had to think about it.

“I knew right away what the answer was. I didn’t have to think about it.” (Teacher, Mobile Phone)

“I looked at my résumé and the timeline.” (Bachelor’s degree, Tablet)

“I thought about the jobs that I’ve had since 2012.” (Additional degree, Computer)

3. B18DEMPLOY01

• All teachers interviewed reported the name of their current employer as the school system because the school system is the entity that sends their paycheck and it is the employer name they report on their W-2.

• Several respondents had trouble entering the name of their current employer because they reached a character limit. RTI should consider an unlimited character text box to ensure accuracy.

“The district employs us. The school has a common name and doesn’t signify where I work.” (Teacher, Computer)

“The school is the building I work in. The school system is who hired me, who pays me.” (Teacher, Computer)

“I put the school system because it is the umbrella where I work and is what is on my W-2 Form.” (Teacher, Tablet)

4. B18DEMPZIP01

• Many say it is easy to remember their zip code. Some say it is especially easy to remember if their employer’s zip code is the same as their home zip code.

• However, some cannot remember the zip code and had to look it up either on their résumé or via a Google search.

• One respondent who is self-employed and travels to different locations to see clients was unsure what zip code to use. Another respondent had multiple positions at different zip code locations for the same company. She was not sure which zip code to enter.

“Because I just started working there, it is difficult.” (Teacher, Computer)

“If I didn’t live in the same zip code as the main office, it may have been more difficult.” (Teacher, Mobile Phone)

“I had to think about this one a bit because I travel to my clients. So, do I put my home address or what?” (Bachelor’s degree, Mobile)

“Very easy. No problem at all.” (Additional degree, Computer)

5. B18DSTART01

• Most say it is easy to provide this information, especially since they can reference either their résumé or LinkedIn profile.

• One teacher was unsure whether to use:

• The date she was given the job offer

• The first date of school

• The date of her first paycheck

• Some say it’s helpful to think about when employment occurred relative to other important milestones, while others do not find this helpful.

“Easy. I like that they include the word ‘first’ since I’ve been employed twice by them.” (Teacher, Computer)

“The year was easy but the month was more difficult. I had to look at my résumé.” (Bachelor’s degree, Tablet)

“It was helpful to think about when I got the job in relation to different political elections.” (Bachelor’s degree, Mobile)

“It doesn’t really help me remember other milestones.” (Additional degree, Computer)

6. B18DWKMON01

• Respondents with non-traditional jobs who are not employed 12 months of the year have some trouble answering this question.

• Some don’t know whether to count partial months.

• Some teachers who do not get paid during the summer are not sure whether to classify summer break as “unpaid.”

• A few say that the question initially seems a little overwhelming because of all the boxes that appear on the screen. It is helpful that participants can “Select All” and then deselect months they did not work.

• Most with more traditional jobs are confident in their answers.

• No respondents have suggested improvements to question formatting.

“The question wasn’t that difficult. The only thing that’s weird is the summer months.” (Teacher, Computer)

“It’s a lot to figure out at first. There are lots of boxes so it’s a lot to look at initially. It helps that you can ‘Select All.’” (Teacher, Computer).

“It was difficult for me because I wasn’t employed for the entire timeframe. I wasn’t working for them full-time, but I did contract work for them.” (Bachelor’s degree, Tablet)

7. B18DEMPBRK01

• The only additional reasons respondents think should be included are “company unable to pay salary” and “graduate student employment.”

“In some school districts they might not have a job this year, but they might the next due to ebb and flow or budget.” (Teacher, Computer)

“It needs to take into consideration graduate student employment.” (Teacher, Computer)

“Add something about the company being a non-profit and they are unable to pay my salary.” (Bachelor’s degree, Tablet)

8. B18DEMPLOYA01

• Some would prefer to enter their salary separately from bonuses, tips and commission. Note that several respondents say that the salary number that is typically top-of-mind does not include bonuses, tips and commission.

• Some suggest that the question clarify whether “average hours per week” is “total hours worked” or “hours paid for.” In other words, many respondents only get paid for a 40-hour work week, but they put in well over 40 hours of work on average and they’re not entirely sure how to respond to the question.

• Respondents with non-traditional jobs have some trouble answering this question.

• A few former k-12 teachers who are now in graduate school and have “teaching assistantships” get paid a lump sum bi-weekly. They found this question difficult to answer because they had to convert their pay into “weekly” or “monthly” amounts to meet the question requirements.

• Note that it is harder for respondents to remember this information for previous jobs than for their current/most recent job.

• Note that one respondent who works for the government says it can be difficult to know your exact salary without looking it up online through the government pay system.

“They should have a bi-weekly option. As a teaching assistant, I get a lump sum payment every 2 weeks so I had to calculate what that would be ‘weekly’ or ‘monthly’ to answer the question.” (Teacher, Computer)

“I think there should be a checkbox to indicate seasonal work. The school year is only 10 months so I only get paid for part of the year, but this makes it look like I work year-round.” (Teacher, Computer)

“They should separate out bonuses, tips and commissions because that’s not part of a base salary and may give the wrong idea about how much I make.” (Bachelor’s degree, Tablet)

“Is this the average hours per week that I work or that I am paid for? They are very different.” (Bachelor’s degree, Computer)

“This was harder to remember the further back I go. I don’t really remember that well so this is a guess.” (Bachelor’s degree, Computer)

“What about contract work? I work hours as I get them, but even my hourly rate changes based on each contract.” (Bachelor’s degree, Tablet)

9. B18DEMPLOYC01

• Same as Probe 8.

10. B18DOCCEX01

• Those with more specific job titles or unique job specialties have trouble finding their title in the dropdown box. Also, those with multiple job titles within their company can’t find something that represents them. Many say the dropdown is too limiting and they would prefer to enter an open-ended response of their own.

• Former K-12 teachers who are now graduate students holding teaching assistantships are not entirely sure how to answer because their job might encompass several different job titles.

• Some say the dropdown choices are overwhelming.

• As in previous waves, mobile respondents have trouble with the programming of this question because their mobile keyboard pops up and covers the dropdown box, which prevents them from selecting the appropriate response.

• Note that if a respondent has had multiple titles within the same company, they tend to be unsure which title to enter. The question text should specify whether respondents should enter their “current” job title or some other job title.

“It is a little tricky to find the right category to select. There are a lot of options here.” (Teacher, Mobile Phone)

“Positions change from year to year. It would be helpful to have the question specify if this is for my current position or my past position.” (Teacher, Computer)

“My position is described as a Teacher Librarian and that wasn’t there. I still consider myself a teacher, but in the core of my position I am a Librarian. So, I chose Librarian. I think it is mostly the same. I am an Elementary Librarian, but their option misses the ‘teacher’ part. That’s an important part to me.” (Teacher, Computer)

“My position is very specific so I had to choose something that is close, but not exactly right.” (Bachelor’s degree, Tablet)

“My position is Program Management and Implementation, but there wasn’t a choice for that so I had to choose another one and just go with it.” (Bachelor’s degree, Computer)

“The categories are a little confusing. There are so many of them.” (Bachelor’s degree, Tablet)

“I have multiple positions in the same company, so my duties include other things that are not related to the title I selected.” (Additional degree, Computer)

11. B18DEMPTYP01

• Respondents do not have trouble answering this question. The answer choices are clear.

12. B18D1IND01

• In their own words, industry means:

• On a global scale, what is your company producing?

• Type of field/category/section the employer is involved in

• Primary mission of the company

“The field or profession that you work in. The career you’re in.” (Teacher, Computer)

“I work for a school so it is definitely education.” (Teacher, Computer)

“The area or type of company or institution that you work for.” (Teacher, Computer)

“The area of the job market the company is in.” (Bachelor’s degree, Mobile Phone)

“It is the area or field of work your profession is in. It can encompass several jobs and occupations.” (Additional degree, Computer)

“The classification, background and primary mission of the position.” (Additional degree, Computer)

13. B18D2IND01

• The first answer choice, “Professional, scientific, and technical services” is very broad. In addition, some don’t understand why “professional” is grouped together with “scientific and technical.” For example, “Financial and insurance” is also “professional.”

• Some respondents are surprised their industry is not listed since they believe it is common (e.g., transportation, property management, etc.).

“Public Transportation is the service we provide, so I selected ‘Something else’ and typed it in because it wasn’t there.” (Bachelor’s degree, Computer)

“‘Professional’ is too broad of a category. A lot of different things can fall into that category.” (Bachelor’s degree, Computer)

14. B18DJBRESP01

• Teachers are unsure whether “supervise the work of others” should include their students.

• Most understand that “supervise the work of others” means overseeing the work of others, being a mentor, being there to answer questions, and conducting performance reviews.

“Does supervising other students count? I supervise student assignments, but I don’t supervise the work of colleagues.” (Teacher, Computer)

“To me, supervising others means going over any type of work that assistant teachers or students give me. I have to make sure that everything is correct and up to date.” (Teacher, Computer)

“It means, ‘Am I helping, teaching or guiding subordinates?’” (Teacher, Computer)

“What’s tricky here is that I should have said no. Do they want to know about other professionals or kids? I supervise kids not professionals.” (Teacher, Computer)

“It means ensuring people adhere to department policies, managing the interns, submitting my reports on time and making sure everyone submits their reports on time.” (Bachelor’s degree, Mobile Phone)

“To me, it is doing reviews, managing the other workers, having someone who reports to me and delegation of work.” (Additional degree, Computer)

15. B18DAUTONM01

• Several respondents have trouble answering this question because they feel the proper response is a combination of the answer choices provided. Some suggest it would be easier to answer the question if it was on a scale with the scale anchors being “Someone else decides what I do and how I do it” and “I am basically my own boss.”

“I am deciding between the middle two answer choices because my schedule is set by someone else and I have no say in that, but I have the freedom to pick my lessons and books within the curriculum.” (Teacher, Computer)

“The superintendent and principal come up with goals for each grade. These goals need to be achieved by a certain marking period so I take the goals and plan it out in my classroom lessons.” (Teacher, Computer)

“I think the question should be on a sliding scale because my answer would be a combination of the answer choices.” (Bachelor’s degree, Computer)

“I just thought about where I am on the totem pole of seniority. Since I am the overall manager for some of my shifts, I have a little more freedom than I would otherwise.” (Bachelor’s degree, Mobile Phone)

“I thought about what my day to day tasks are like and what is most appropriate to what my job is.” (Additional degree, Computer)

NON-PROBE QUESTION ABOUT TELECOMMUTING

• One teacher was not sure what telecommuting is. This might be a case where RTI could provide hyperlinked help text.

16. B18DNSFA01

• Some assume that a bachelor’s degree is required, but they cannot remember it being specifically stated in the job description.

• Some do believe it was stated in the job description.

“There are alternate paths to becoming a teacher and with the shortage, you can get to be a teacher without getting a degree at first. But then you have to take classes to get certified while teaching.” (Teacher, Computer)

“I am a graduate teaching assistant, so no one can get this position without having first gotten a bachelor’s degree.” (Teacher, Computer)

“It is not explicitly stated in the job description, but I just know that you wouldn’t be considered if you didn’t have a Bachelor’s Degree.” (Bachelor’s degree, Computer)

“In the job description, it says a bachelor’s degree was required.” (Additional degree, Computer)

17. B18DCHNG101

• Not enough respondents received this probe to provide feedback.

18. B18DCHNG201

• Not enough respondents received this probe to provide feedback.

19. B18DEMPDEBR

• Yes, respondents say the employment information they provided is accurate.

• Not everyone’s résumé contains the same employers they reported in the interview because:

• The interview only goes back to July 2012 and some have employment history before then.

• Some do not have their current employer on their résumé yet.

• Respondents who have trouble reporting their job title in the interview say that the title they reported does not match what’s on their résumé because the interview answer choices were not specific/exact enough.

• One respondent who has held multiple different positions within the same company felt like the survey did not capture that information.

• Some say they would like it if they submitted their résumé in advance and the survey uploaded the information for them. Other respondents don’t really seem to care one way or the other.

“My résumé doesn’t have my current job because I haven’t updated it yet.” (Teacher, Computer)

“It’s not the same as what I said here because I told you the school system, but on my résumé, I have the name of the actual school I work in.” (Teacher, Computer)

“I have more employers on my résumé because I have jobs that go past July 2012.” (Bachelor’s degree, Computer)

“Having my résumé uploaded ahead of time would be nice, but only if the upload works well and is correct. If I have to spend time correcting it, it would be a waste of my time and not worth it.” (Bachelor’s degree, Mobile Phone)

“I would appreciate it if we could have all the information automatically uploaded.” (Additional degree, Computer)

20. B18DNOWRK

• Respondents decide on their answer by referencing their résumé and thinking back on what they were doing during this timeframe.

21. B18DCARMLT

• This question is a little ambiguous for some participants. For example, if you have worked for the same employer, but in two different departments, is that one career or two?

• Most respond by counting the number of industries they’ve worked in.

• The question after this probe asks about “length of their career.” Some are unsure whether or not to include internships and volunteer positions.

“The part-time jobs were just to help me get by, but I don’t count them as careers.” (Teacher, Computer)

“A career is a thing you work at for a long time. Jobs are not.” (Teacher, Computer)

“I thought about all the jobs that I’ve had in the past years and the other fields I was in.” (Bachelor’s degree, Computer)

“I answered just by looking at the different industries that I’ve been in and worked in.” (Additional degree, Computer)

22. B18DNEGOTIAT

• This question is clear for non-teachers (although those who work for the government say they really can’t negotiate much).

• Some teachers have trouble with this question because:

• Their salaries are set and non-negotiable.

• They are a member of a teachers’ union and the union negotiates salary on their behalf.

“Teachers have set salaries that are determined by teaching experience or number of years in the field. You are not allowed to negotiate.” (Teacher, Computer)

“I have never negotiated. Once you’re in the government, it’s almost impossible to negotiate.” (Bachelor’s, Computer)

“I always try to negotiate when I start a new job.” (Additional degree, Computer)

23. B18DEMPFINAL

• Questions that are most difficult to answer include:

• Whether they have had more than one career

• Starting and ending salary for past jobs

• How much freedom they have in deciding what work they do and how they do it.

• Respondents feel fine about providing this level of detail. However, as in previous waves, some respondents want more information about the purpose of the survey and how their information will be used to feel more comfortable responding. One respondent specifically mentioned that she wants to know more about respondent confidentiality and if her answers will be tied back specifically to her, or whether her answers will be kept anonymous.

“I was not totally comfortable answering all the questions, but I guess whatever they need.” (Teacher, Computer)

“It’s fine, but I would want to know what the survey is for and how the information is being used. I work for the government so I would have to be careful with my information and how I answer the questions. I would want to know about the confidentiality of my responses.” (Bachelor’s degree, Computer)

24. B18EANYTCHX

• Definition of a regular teacher:

• Has an educational background/teaching degree

• Someone who works full-time with students

• Is the primary teacher in the classroom

• Spends time outside of the classroom preparing for the week

• Non-traditional educators (e.g., those in AmeriCorps or Teach for America) are not sure how to answer this question because they teach in the classroom, but they are not necessarily a “regular teacher.” Note that these types of educators seem unsure how to answer many of the questions in the teaching section of the survey.

“My role in the school was a strange one. I’m not sure if it fits for me to answer ‘yes.’ I had specific classes I visited and taught. I was permanently on staff at the school, but I didn’t grade papers or give tests. But I did meet parents and run after school programs. It was a hybrid role that’s not like a traditional teacher. There are so many programs now-a-days like AmeriCorps or Teach for America and I’m not sure how people in those situations would answer.” (Teacher, Computer)

“I worked as a volunteer teacher in a classroom, but that’s not how they’ve defined it here. It’s not my career it’s something I do sometimes. A regular teacher is someone who’s invested a lot and this is what they do for a living. They are dedicated to teaching.” (Teacher, Computer)

“A regular classroom teacher is someone whose name is on the students’ report cards and he/she is responsible for all students in the classroom and for all classroom decisions including grading, lesson plans and contacting parents.” (Teacher, Computer)

“I believe that the definition of a regular classroom teacher changes as you move from kindergarten to the 12^th grade. It goes from being about general education in the younger grades to being knowledgeable and teaching one specific subject.” (Bachelor’s degree, Mobile Phone)

25. B18EANYTCH

• Preference between the two question formats is split.

• Some are unfamiliar with the terms:

• Itinerant teacher

• Don’t know the difference between “support teacher” and “teacher’s aide”

• Some are unsure what would be classified as “other” (Note that one respondent who teaches gym and is a coach selected “other.”)

• Suggestions for improvement:

• Non-teachers would like definitions of each answer choice because they are unfamiliar with some of the terms.

• One respondent suggested including something like “auxiliary education roles such as teacher observer, classroom monitor, etc.” because these are important roles in education even though they’re not classroom teachers.

“I’m not sure how to classify a non-traditional teaching role like AmeriCorps or Teach for America. You might need to have different questions for these people. You may want to add ‘Other auxiliary roles that are education related,’ which could include things like teacher observers, classroom monitors, etc. These are important roles, but they’re definitely not teachers.” (Teacher, Computer)

“The first way was very simple and straightforward.” (Teacher, Computer)

“I like the second way because it’s more inclusive and collects more detailed information. It gives you more options to choose from and you can check more than one response. You get a more accurate answer this way. It would be hard for someone in a non-traditional teaching role to respond.” (Teacher, Computer)

“I don’t know how they distinguish between Teacher’s Aide and Support Teacher. I don’t really know the difference myself.” (Teacher, Mobile Phone)

“I like the second way because it was a lot more specific, was less confusing and allowed multiple answers if needed.” (Bachelor’s degree, Tablet)

26. B18ETHNKINFL

• Some respondents think this question should be asked on a scale to capture more nuance.

• Other factors that should be included:

• Family history of teaching

• Looked up to a childhood teacher as a role model

• Separate “subject” and “location” because they are totally different things

• Work/life balance

“I think you need a better scale for this question, not just ‘positive,’ ‘neutral’ or ‘negative.’ It needs a 5-point scale because you need more nuance.” (Teacher, Computer)

“I think it would be better asked in a ranking such as ‘Which was most important and which were other deterrents?’ Or, ‘To what degree did these things impact your decision to become a teacher?’ Maybe on a scale from 1 to 5.” (Teacher, Computer)

“Subject and location should be split, they don’t belong together.” (Teacher, Computer)

“They need to add something about ‘relationships to other teachers,’ whether they be teachers I personally had or family members who were teachers that influenced me.” (Teacher, Computer)

“They should add ‘my own personal education experience,’ because I knew I didn’t want to be like the type of teachers I had growing up, I wanted to be different.” (Teacher, Computer)

“One thing that I considered is that I wanted a career that would allow me to align my schedule with my kids, once I had them. Teaching does that perfectly.” (Teacher, Computer)

27. B18EINT

• Most think the text in the parentheses is unclear and needs to emphasize more clearly that this question is not about student teaching.

“I had to read this a couple times to really understand what the question was asking.” (Teacher, Computer)

“The part ‘result in a regular teaching certificate’ was confusing to me.” (Teacher, Computer)

28. B18EPRP

• Some respondents think this question should be asked on a scale to capture more nuance.

• Additional experiences not represented in the list:

• How to deal with demographic diversity

• Working with co-workers and administrators

• Familiarity/comfort with the subject area

• Mental and emotional stress training (e.g., training for teachers to deal with students who have mental disorders or emotional stress and training for teachers to deal with their own emotional stress of managing a classroom)

• Assigning grades to students

• Scheduling homework assignments

• Working within the educational standards

• Documentation procedures

• Emergency training

“Any first year teacher who says you’re adequately prepared is a liar. We could be here for the rest of the day if you want me to add what’s missing to this list. But there should definitely be something about ‘mental and emotional training’ both in terms of how teachers deal with students who have mental or emotional issues and how teachers deal with their own mental and emotional stress as well.” (Teacher, Computer)

“This list could be endless! Add assign grades to students, schedule homework assignments, engage in collaboration with other teachers, engage with administration, work with parents and the community should be two separate things, work within the education standards, dealing with diverse populations.” (Teacher, Computer)

“I would add something about familiarity and/or comfort with the subject. I got lucky because I was assigned a subject that I know a lot about and actually studied, but I know some other teachers who just got placed in a subject and they didn’t necessarily have a lot of knowledge about it or feel comfortable teaching the kids the subject.” (Teacher, Computer)

“I think it would be better to have a scale of 1 to 5. Some I would have said ‘A little bit’ or ‘Partially’ whereas here I was forced to go to one side or the other even if I didn’t think it was completely accurate.” (Teacher, Mobile Phone)

“The biggest thing teachers struggle with in their first year, in my mind, is time management, mental health, mental stress and stress management because that usually is a huge problem. I know I wasn’t prepared for that.” (Teacher, Computer)

29. B18EHLP

• Some respondents think this question should be asked on a scale to capture more nuance.

“I don’t think ‘yes/no’ is really sufficient. I think it would be much better to have a 1 to 5 scale and ask something along the lines of ‘How much did the school district help you?’” (Teacher, Computer)

“Same comment as the last question about having a scale because they kind of helped me during the first year of teaching, but not much. I just picked ‘no’ because I didn’t have any other options to select something that would have been more accurate.” (Teacher, Computer)

30. B18ECRTDT

• Most do not have trouble answering this question because it happened the same year they graduated college.

• After Probe 30 the survey asks, “What grades are you certified to teach?” Teachers who have certifications in multiple subject areas for different grade levels have a hard time answering the question. Also, some teachers cannot select the appropriate grade levels if their certification does not match the grade level groupings in the survey.

“I had to check k-5 because that was the only option that included 5th grade, but I’m actually only certified to teach 5th grade and up.” (Teacher, Computer)

31. B18EJBSL

• Some non-traditional educators may teach at more than one school at any given time. They are unsure how to answer this question.

• Those who work at schools with a common name find it difficult to find their specific school because there are too many options with the same name to scroll through. They suggest being allowed to search by city and state first, then school name.

“This was difficult because the name of my school is always listed different ways so I just had to try a few times until I typed it in so that it was recognized.” (Teacher, Computer)

“I couldn’t find my school, maybe because it’s a religious school.” (Teacher, Computer)

“I liked the search being narrowed down as I typed, but it wasn’t the easiest for me because my school has a very common name and there must be at least 50 other schools with this name.” (Teacher, Computer)

NON-PROBE QUESTION “WHAT SUBJECTS DID YOU TEACH?”

• Some respondents have difficulty answering because they teach across multiple different subjects (e.g., how health relates to history class or science class, etc.).

• Some teachers are unsure how to classify subject areas they teach after school since they’re not technically getting paid to teach them (e.g., tutoring ESL after school).

• Teachers are not sure whether the question refers to the one school they just entered the survey or whether they should consider all the schools they’ve worked for.

“This is too broad of a question. I teach multiple subject areas so I am checking them, but I am not necessarily an expert in all of them.” (Teacher, Computer)

32. B18EJBGR

• Some respondents do not understand what “ungraded students” means and think it needs an explanation. They said they’ve never heard this term before. Others glance at the word and assume it says “undergraduate” and then get thrown off when they check the box and it deletes the response in the dropdown boxes that they have already selected.

• This is another example of a question where teachers are unsure whether to consider the grade levels they taught only at the most recent school they just entered the survey, or whether they should consider the grade levels they’ve taught throughout their entire teaching career.

• One respondent teaches pre-school, which is not an option on the list of answer choices.

“Are they asking for the grade levels I taught specifically at the school I just mentioned or ever in my teaching career?” (Teacher, Computer)

“What is ‘ungraded students’? It needs an explanation. When I first read it, I thought it said, ‘undergrad students’” (Teacher, Computer)

“Add in ‘for this school year’ if that is what they are actually looking for. This question made me think back to previous years.” (Teacher, Computer)

“I teach pre-school as well as kindergarten. Since pre-school isn’t listed I am just going to put kindergarten, but this isn’t an accurate answer for what I actually teach.” (Teacher, Computer)

33. B18EPRTIME

• Most teachers are not confident in their answer. They have no idea how long the principal has held the position. One teacher was not sure whether the question wanted him/her to consider how long the principal was there in total or just while that teacher was employed at the school.

“I have no idea, I had to guess. Are you asking how long my principal held the position in total even before I got there or just while I was there? I have no idea either way.” (Teacher, Computer)

“This is tricky because we had a principal who left and then we got an interim administrator and she had the position until we got the new one. Which do you want me to answer for? I can answer the interim one easily.” (Teacher, Computer)

34. B18EPRSUPP

• Teachers say this question is easy to answer since it uses a scale.

• A few teachers are unsure which principal to consider in their response to the question since they had more than one at their most recent school.

• One teacher said the question is difficult to answer because responses could vary based on the situation.

“This was hard for me because I wasn’t sure which principal they were referring to. Is this question related to the previous question? Which principal are they asking about?” (Teacher, Computer)

“This question was a bit difficult for me because there are so many different situations to think about. I think this is a situational thing where this can change based on what is going on and how the principal reacts.” (Teacher, Computer)

35. B18EUNION

• Respondents are confused by the phrase “in a bargaining contract” and are unsure what this means.

• Suggested changes to question wording:

• “Are you a member of a teachers’ union?”

• “Do you pay dues to a teacher’s union?”

“The part that says, ‘in a bargaining contract’ was confusing to me and threw me off.” (Teacher, Mobile Phone)

“I had a little bit of difficulty answering this question. I was a little confused if it’s asking if I’m part of a union or not.” (Teacher, Computer)

36. B18ELVRA

• Other reasons that should be included:

• Retired

• Found better opportunities outside of teaching

• Left for graduate school or to pursue a higher degree

• Respondents believe you should be allowed to choose more than one answer and that you should be able to explain your answer further if needed.

“I think there should be a place to explain because I was actually really happy during most of my time there and I feel like this doesn’t fully represent what I felt. I was only dissatisfied the last year I was at the school. I liked the principal that we had, but he left and the new one wasn’t good, so I left.” (Teacher, Mobile Phone)

“One option that is missing is that I left for graduate school to pursue a higher degree.” (Teacher, Computer)

37. B18ELVRB

• Other reasons that should be included:

• Being micromanaged

• Overall level of collegiality

• Staff morale

38. B18ESCHLEVA01

• Not enough respondents received this probe to provide feedback.

39. B18ETCHSAT

• Some were not sure whether the question was asking about their current school or a previous teaching position.

• Options missing from the list:

• Work environment

• Professional development

• Overall level of collegiality

• Staff morale

• Some respondents think this question should be asked on a scale to capture more nuance.

“I don’t think the ‘yes/no’ answer choices are good because it is very broad and could be more specific with a scale. I guess this is forcing you to answer these as ‘Ultimately, am I satisfied or not?’” (Teacher, Computer)

“Is this asking about my current school or my old school? Because they are different. I am at a public school now, but I was at a private school.” (Teacher, Computer)

40. B18ETCHSTAY

• Options missing from the list:

• Salary and benefits

• Opportunities for advancement

• Intellectually stimulating work

“Intellectually stimulating work. I feel that I am good and effective at my job. I also like that I am intellectually challenged and can be creative in my work.” (Teacher, Computer)

41. B18EMOVE

• Respondents think about their current job satisfaction and their experience with administrators.

“I’ve thought about becoming a principal before. I am happy with my position now, but someday I may become a principal.” (Teacher, Computer)

42. B18ETCHLEVA

• Participants did not have any suggestions for additions to the answer choices. They feel the main reasons are covered and anything that might be missing can be addressed by selecting “another reason not listed.”

43. B18ETCHLEVB

• Not enough respondents received this probe to provide feedback.

44. B18ETCHGRT

• If respondents are confident that they have never heard of the TEACH Grant Program, then the help text is not very helpful.

“The help text didn’t matter because I already knew I hadn’t heard of the grant. The help text is pointless.” (Teacher, Computer)

45. B18ELNINCT

• Respondents who participated said “no” to this question. Many say they learned about the loan forgiveness program after they began studying to be a teacher so there was no way it could have influenced them to pursue a career in education.

“I went into teaching for other reasons.” (Teacher, Computer)

“I didn’t know about this program prior to starting my teacher education program so it couldn’t have influenced me.” (Teacher, Computer)

“The question is straight forward. I learned about loan forgiveness after I became a teacher, so it wasn’t a factor when I was deciding to pursue a career as a teacher.” (Teacher, Computer)

46. B18ETCHFINAL

• Teachers think the yes/no questions are difficult to answer because they don’t capture enough nuance.

• One teacher points out that answers to the questions in this section could be very dependent on state-of-mind/emotional state at the time when taking the survey. For example, one’s answers might be very different if taking the survey soon after teaching a particularly difficult/rowdy group of students versus one’s favorite class.

“A lot of the questions were opinion based, which can change depending on the state of mind or the emotional state I am in when I am taking the survey. Like right now, I am in a great mood because I just finished the day with my favorite group of students. Last year, I probably would have answered these questions differently because my last class of the day on Thursdays was terrible.” (Teacher, Computer)

“The ‘yes/no’ questions could go into more detail or be more specific.” (Teacher, Computer)

47. B18AFINCON

• One respondent is unsure whether having a roommate meant he should select “yes.” You share bills with a roommate, but not income and budgeting.

• One respondent was uncomfortable with this line of questioning.

“I think this is weird because I live with my sister who owns the apartment so I pay her rent, but we are not financially intertwined. The help button was very helpful in this instance.” (Teacher, Computer)

“Are they talking about personal finances or like a roommate sharing rent? The wording is awkward.” (Teacher, Computer)

“I feel weird. I don’t like answering questions about who lives with me and who pays what.” (Teacher, Computer)

48. B18AFINWHO

• Respondents do not have trouble answering this question.

49. B18AHCOMP1

• Respondents think this question is comprehensive because you can select “another person.”

50. B18AHCOMP2

• Respondents do not have trouble answering this question.

51. B18FSEX

• Most respondents are fine answering this question, although they recognize that others may feel uncomfortable.

“It doesn’t bother me, but may be people that it would bother.” (Teacher, Computer)

52. B18FGENDERID

• Unfamiliar terms:

• Genderqueer or gender nonconforming

• Some are not sure what would be classified as “a different gender identity”

• Terms that should be included:

  • Gender fluid

  • Non-binary

  • Hermaphrodite

53. B18FLGBTQ

• Terms that should be included:

• Asexual

• Prefer not to answer

54. B18FDISCRIM

• Some say the explanation of discrimination helps, and others say they don’t need it. Many say they believe they could answer the question without the explanation.

• Respondents decide on their answer based on personal experience.

• One respondent suggests that “age discrimination” is missing from the list.

• Most respondents feel fine answering the question. However, they guess that those who have experienced discrimination might feel uncomfortable.

“I think the word can be interpreted differently. It is helpful to have the definition according to the person who is asking the question so I know what they are looking for.” (Teacher, Computer)

“I thought about my own experiences in the workforce and the fact that I definitely have definitely witnessed ‘white male privilege.’” (Bachelor’s degree, Mobile Phone)

55. B18FACCEPT

• This is another example of a question where it is unclear to some participants what “workplace” or which job/position the question is referring to. Is it their current workplace and, if so, which job/position at their current workplace?

• “Accepting” in the workplace means:

• Not talking about others behind their back

• Not making fun of others

• Letting people be who they are

• Treating everyone with respect for their qualifications, not their personal life

• Open to everyone, make everyone feel comfortable and welcome

• Make a point that policies and procedures apply to ALL people regardless of sexuality, race, etc.

• Hire based on skill set

• You are not met with resistance for being who you are

• Being comfortable and feeling it’s a good working environment for all, regardless of their differences. “Accepting” does not mean the same thing as “discrimination,” which involves not being able to advance your career.

“A workplace that allows you to be open, be who you are, and treats your ideas as equal to anyone else’s. Is this question asking about my current workplace or my work experiences in general?” (Teacher, Computer)

“Making sure everyone is educated in these terms and aware and respectful that we are all people first. Not an environment where you get singled out because of your differences or where job performance or hiring decisions are based off that. If it is out in the open, then it should be celebrated just like everyone else.” (Teacher, Computer)

“I work for the government. So…I felt comfortable in the last administration that this would be a good work environment for anyone who’s different. In this new administration, there’s a preference for white Christians. There might not be blatant discrimination, but I feel that certain people would not be as accepted based on the language I hear people using.” (Bachelor’s degree, Computer)

“You can be who you are without having to hide part of your identify. You are not met with resistance in any way and you are comfortable walking into the building you work in.” (Bachelor’s degree, Tablet)

“The workplace makes it a point to emphasize that policies and procedures apply to all people regardless of sexuality, race, etc.” (Bachelor’s degree, Computer)

“A workplace that is open to everyone and making everyone feel comfortable and welcome.” (Additional degree, Computer)

“Having protections in the workplace so discrimination doesn’t happen.” (Additional degree, Computer)

56. B18FDEPSAM

• Several respondents find this question confusing. Other than “foster care” and “adoption,” they’re not sure what else would qualify.

• One respondent suggests asking “Are you financially responsible for any of your dependents through…”

• Adoption (yes/no)

• Foster care (yes/no)

• Step children (yes/no)

• Etc.

“I guess if I adopted any kids. I don’t know what else would qualify other than foster care or adopting.” (Teacher, Computer)

“It needs to be asked a different way, but I am not quite sure how to phrase it.” (Additional degree, Computer)

“I guess it’s asking, ‘When did you financially become responsible for the child?’” (Additional degree, Computer)

57. B18FCSTDYCR

• Respondents are very confident answering this question.

58. B18FLVCHLD

• Respondents think about if they got paid during their leave.

59. B18FDLVCT

• Respondents say it’s easy to answer the question.

• Some male respondents say it might be helpful to add “days” and/or “weeks” since they rarely take full months off for the birth of a child.

“It was ok. I think people will round.” (Bachelor’s degree, Tablet)

“The jump from days and weeks to months and years was a little tough.” (Bachelor’s degree, Computer)

60. B18FOTHER

• The question is asking whether respondents support anyone else financially by 50% or more. Some respondents think about who they can claim as a dependent on their tax return.

“It’s asking if someone is solely dependent on me and my spouse. We’re helpful, but we don’t fully support anyone else.” (Teacher, Computer)

“It’s a tax question. They want to know about my tax dependents.” (Bachelor’s degree, Computer)

61. B18FSPAMT

• Respondents’ level of confidence in answering this question is varied. Some feel confident, while others have to make an educated guess.

62. B18FSPOWE

• While some respondents struggle to provide an accurate answer in Probe 61 because the question asks for a specific loan amount, they find it much easier to answer Probe 62 since it’s a multiple-choice response question and they know if their spouse is still paying off their loan.

“I’m pretty confident on the amount still owed, but not very confident on the total loan amount.” (Teacher, Computer)

63. B18FSPLNPY

• Respondents’ level of confidence in answering this question is varied. Some feel confident, while others have to make an educated guess.

64. B18FSELLPO

• Some respondents feel fine answering this question, while others have a slightly negative reaction either because it’s “depressing to think about,” or they feel it’s very private information that they’re being asked to share.

• Most agree that this is an eye-opening question that they have to really think through before providing an answer.

• One respondent suggests adding a “Prefer not to answer” option.

“I don’t like answering this question. I never speak about these issues with anyone other than my husband.” (Teacher, Computer)

“I don’t feel it is a relevant question to everything else asked, but it was easy enough question for me to answer.” (Teacher, Computer)

“This question made me laugh because I’m an educator in the U.S. We’re all poor and paying off our loans for at least 10 years.” (Teacher, Computer)

“I am in a different position now than I was a month ago, so I answered differently than I would have before and this gives a different view.” (Bachelor’s degree, Computer)

65. B18FSTRESS

• Respondents answer this question by thinking about if they are able to pay their bills.

• A few respondents believe that participants with lower incomes might feel uncomfortable answering the financial questions in the survey and would likely want to know why these questions are being asked and how the information is going to be used.

“I had to read it a couple of times. I’ve always been able to pay these things in the past year. The ‘essential expenses’ threw me off, but the examples following it helped to clear things up.” (Teacher, Computer)

“We don’t have any debt, we pay off our bills every month. My comfort level with this question is fine because I have money, but some lower income people will be less comfortable answering and might want to know why this question is being asked.” (Bachelor’s degree, Computer)

“I thought about how much effort I put into my credit score. It was a fairly easy question to answer because I am usually on top of things like bills.” (Bachelor’s degree, Computer)

66. B18FAFFCOST1

• A few respondents do not think the answer choices should be fully randomized and that it makes more sense to group similar responses together (e.g., delayed buying a home/getting married/having children should not be separated by randomization).

• Categories to include:

• Travel or vacation

• Buying a car

• Taking out additional credit cards to assist with payment

• Join the military

“I would include traveling because that is a desire for most people.” (Teacher, Computer)

“I think the ‘delayed’ choices should be at the bottom and clustered together as opposed to randomized.” (Bachelor’s degree, Computer)

67. B18FAFFCOST2

• Most do not have comments about the layout of this question. However, one respondent suggests using bold or underlined text for “cost of undergrad education” to call more attention to this part of the question.

• Categories to include:

• Move in with parents or other family member

68. END

• Most respondents feel fine answering these questions especially via an online survey (as opposed to an in-person interview).

APPENDIX 1: RESPONDENT PROFILE

Note: Numbers may add to more than the total sample because respondents could select more than one response.