Appendix C - Cog Lab Results Report
Appendix C - BTLS W5 2012 Cog Lab 2011 Results Report.docx
Beginning Teacher Longitudinal Study (BTLS) 2009-2012
Appendix C - Cog Lab Results Report
OMB: 1850-0868
Cognitive
Testing
of Proposed Items for the
Beginning Teacher
Longitudinal Study
Summary of Findings and Recommendations
August 2011
Submitted to:
United States Census Bureau
Submitted by:
Macro International Inc.
11785 Beltsville Drive, Suite 300
Calverton, MD 20705
Background
In the summer of 2011, the Census Bureau contracted with ICF Macro, a research and evaluation firm in Calverton, Maryland, to plan and carry out a series of cognitive interviews with current and former teachers. The purpose of these interviews was to gather feedback on a number of proposed items for the surveys that will be a part of the Beginning Teacher Longitudinal Study (BTLS). The BTLS is a national study conducted by the National Center for Education Statistics (NCES) and the Census Bureau. The survey will be administered annually for 10 years to a cohort of teachers in order to gain insight into teachers’ responses to transitions.
This report summarizes the methodology used in conducting the interviews, as well as feedback ICF Macro received from participants about the survey items.
Summary of Methodology
Recruitment Protocol and Summary of Participants
ICF Macro conducted cognitive interviews with nine current or former teachers. Teachers were recruited from several sources, including a list purchased from an outside vendor and an ICF Macro database of educators. An email was sent out to potential participants that described the study and offered an incentive to those who participated. Interested teachers who contacted ICF Macro by telephone or email were then screened and scheduled for an interview.
Depending on their circumstances, teachers were classified as “Movers,” “Leavers,” or “Returners,” and sent the appropriate interview protocol. “Movers” were defined as teachers who had left one school and moved to another within the past five years. “Leavers” were defined as teachers who had left the teaching profession in the past five years. “Returners” were defined as teachers who had left teaching for at least one year and had returned to teaching in the past five years. In addition, participants were asked whether or not they had served as a mentor teacher, which was defined as an experienced teacher who was assigned to provide support to a newer teacher.
ICF Macro interviewed three teachers in each group (Movers, Leavers, and Returners). Participants represented each of the school levels (elementary, middle, and high) and taught in five different states (Alabama, Arizona, Georgia, Florida, and Virginia). Seven of the nine teachers indicated that they had previous mentor experience.
Interview Protocol
Each interview lasted approximately 45 minutes and was conducted by phone. Prior to each interview, the participants were emailed a copy of the survey items and asked to print them out but not to read them until the interview. The interview protocols were different for each group (Movers, Leavers, and Returners). At the beginning of each interview, the interviewer rescreened the teacher to ensure he or she was eligible for the study and classified in the correct group.
During the interview, participants were asked to answer several BTLS items as they normally would if they were responding to the survey. As they answered each item, they were also asked to “think aloud”—that is, to describe out loud their thought process as they read and responded to each item. After the participant answered a series of items, the interviewer asked several follow-up questions to test his or her understanding and interpretation of the items and response options. The interview protocols are included in this report as Appendix A.
Summary of Participant Feedback and Recommendations
The following section of this report summarizes the results of this study. For each item being tested, we provide the wording and response options, a list of relevant findings, and our recommendations for revisions to the item based on those findings.
Q1 Movers: Indicate the level of importance EACH of the following played in your decision to leave LAST YEAR’S school. a. Because of a change in residence. b. Because I wanted the opportunities offered at my current school. c. Because I was dissatisfied with how some of my compensation, benefits, or rewards were tied to the performance of my students at last year’s school. Leavers: Indicate the level of importance EACH played in your decision to leave your pre-K–12 teaching position. a. Because of a change in residence. b. Because I wanted the opportunities offered at my current school. c. Because I was dissatisfied with how some of my compensation, benefits, or rewards were tied to the performance of my students at last year’s school. Returners: Indicate the level of importance EACH played in your decision to return to the position of a pre-K-12 teacher. a. Because of a change in residence. b. Because I wanted or needed a job/higher salary. c. Because of how some of my compensation, benefits, or rewards are tied to the performance of my students. d. Because I wanted the opportunities offered at my current school. e. Because I was able to maintain privileges based on my seniority/tenure. |
Note: These items were included as a part of a larger set of items, which is shown in Appendix A.
Findings
Overall
When answering these questions, five of the nine participants answered in part based on whether the statements were true or important in general, not whether they had played a part in their decision. For example:
One returner rated “Because I passed the required test(s)” as “extremely important” because it was true, even though she actually passed the tests years before she left teaching and it was not a factor in her decision to return.
One leaver rated “Because I was dissatisfied with how student assessments/school accountability measures impacted my teaching…” as “very important.” However, upon further questioning he said that this was not actually a factor in his decision, but he rated it as “very important” because it was something that he felt strongly about.
One mover who was forced to change schools because of budget cuts rated as “slightly important” the items “Because I was dissatisfied with opportunities for professional development” and “Because I was dissatisfied with workplace conditions.” However, neither was actually a factor in why she left.
At least one participant (a returner) factored into her answers the extent to which each statement might be true for other teachers; for example, she rated “Because I was able to maintain privileges based on my seniority/tenure” as “extremely important” despite the fact that she had moved to another state and this could not have been a factor in her decision to return to teaching.
One participant did not check any boxes for this item, because she felt that none of the items applied to her (she wrote in a reason under “other reasons,” but did not check the box). Another participant left most of the rows blank because they did not apply to her situation. A third initially left most rows blank, but then went back and checked “not at all important” for those rows he had left blank.
One returner noted that (p) “Because I liked the school schedule/calendar” could mean that a respondent liked the schedule or calendar of a specific school, but could also mean that he or she liked the general schedule of the teaching profession (i.e., summer breaks).
One mover found (j) problematic (“Because I felt that there were too many intrusions on my teaching time at last year’s school”). She was unsure whether this was referring to additional non-teaching duties that she was assigned outside of the classroom, or solely classroom intrusions such as assessments.
When asked whether there were any other possible reasons that should be included in the question, two participants thought that an item should be added for teachers who left teaching or switched schools because of budget cuts in their district. One participant mentioned that someone could choose to return to teaching because they missed the “camaraderie” (she rated this as “very important” in her own response). Another participant listed reasons that were already captured in other items (e.g., “lack of administrative back-up”) apparently because he wanted to reiterate their importance in his decision.
“Because of a change in residence” (Movers, Leavers, and Returners)
When asked what was meant by this item, all participants said that it referred to any situation in which the respondent moved his or her residence, whether because of divorce, a change in employment by a spouse, the purchase of a house, or another reason.
Participants were asked which item would be most appropriate if they had made their decision because they wanted a job closer to where they lived (i.e., if their decision had been based on the relative location of their home and job, but they had not actually moved). Four said that “Because of a change in residence” would be the most appropriate response, while one said “Because of other personal life reasons” and another said “Because I liked the school calendar.” Three said that this situation would not fall under any of the current items.
“Because of how some of my compensation, benefits, or rewards are tied to the performance of my students” (Movers, Leavers, and Returners)
Participants generally had a consistent understanding of this item, and thought that it referred to bonuses and other ways in which teachers could be rewarded or punished monetarily based on how their students performed on standardized assessments.
Participants generally did not see a distinction between whether their compensation was tied to student performance and the specific way in which it was tied to performance. While two participants noted that the wording of the item seemed to imply that it was referring to the latter, it did not appear that this distinction would have affected any of their responses.
“Because I wanted the opportunities offered at my current school” (Movers and Returners)
Four participants were asked which item would be most relevant if they had made their decision because they wanted to coach soccer at their new school. Three chose this item, while the fourth said that she would write in this reason under “other factors.”
When asked what kind of “opportunities” this item might be referring to, participants mentioned the following:
Professional development opportunities
A job coaching a sports team or leading a club
A position such as a subject area specialist or department chair
The opportunity to train to be an administrator
Mentoring opportunities
The opportunity to work with technological resources not available in other schools
“Because I decided to take courses or pursue a degree to improve career opportunities WITHIN/OUTSIDE the field of education” (Leavers)
Participants consistently understood the difference between these two items. They indicated that improving career opportunities within the field of education referred to situations in which teachers pursued coursework or a degree to further their careers as teachers or, potentially, administrators. Improving career opportunities outside the field of education referred to teachers who were looking to change careers (e.g., by attending nursing school).
“Because I was able to maintain privileges based on my seniority/tenure” (Returners)
Participants had a consistent understanding of this item. When asked what the item meant, one participant commented that retaining her seniority when she returned to teaching would give her more job security and preference for certain positions (such as that of a department chair). Another commented that it meant that you would not need to “start over” when you returned to teaching.
“Because I wanted or needed a job/higher salary” (Returners)
Returners were asked which item would be most relevant if they had returned to teaching because they were laid off from a non-teaching job. Two chose item (c) “Because I wanted or needed a job/higher salary,” while the third chose item (h) “Because I wanted job security.”
When asked what this item meant, participants consistently thought it referred to a situation in which someone returned to teaching because they had no other job or because they needed more money.
Recommendations
NCES should be aware that some participants will answer this question based on the extent to which the statements were true for them, not the extent to which they influenced their decision. However, it is unclear how this issue could be addressed without drastically changing the question (e.g., adding “I returned to teaching because…” to the beginning of every item).
Consider asking this item as an open-ended question, rather than a selected response matrix. While this may increase transcription and coding costs, it may also lead to more accurate measurement of the reasons teachers make decisions, rather than what is important or true of them in a more general sense.
There are likely to be some respondents who have difficulty answering this question because it does not include a “not applicable” response option. However, adding this option would likely change participants’ responses and affect the comparability of longitudinal results.
Consider adding an item to the survey for movers and leavers who changed schools or left teaching involuntarily because of surplussing or district budget cuts.
Consider either revising the wording for (a) (“Because of a change in residence”). Alternatively, consider adding an item to encompass teachers who made decisions based on their desire to work closer to home, but did not change their residence.
Consider adding examples to the item about opportunities offered at respondents’ current school.
Q2
What are your estimated annual before-tax earnings at this job?
__ less than $40,000 __ $40,000 to $49,999 __ $50,000 or more |
Findings
All but two of the participants said they would respond to this item if it were on a survey. Of the two participants who said they would not, one said he would be more likely to respond if there were only two response options: “less than $X” or “$X or more.” The other participant said she would be more likely to respond to the item if it explained more clearly why Census wanted the information and how it would be used.
Three participants said they found the italicized part under the question somewhat confusing, particularly the direction related to military earnings. None of these participants were in the military service.
One leaver had difficulty answering this item because he was unsure if “this job” referred to the teaching position he left or the position he currently held. He answered based on the non-teaching position he currently held.
Recommendations
Based on findings from testing, it is possible that the response rate to this question might be slightly higher if only two response options were provided (e.g., “less than $50,000” and “$50,000 or more”). However, most participants indicated that they would answer the question as it is currently worded.
For leavers, make sure that in the context of the survey it is clear to which job this question is referring.
If this question is asked of current teachers, do not include the direction about military earnings because it may be confusing and lead people to include other sources of income as well.
Q3
This school year (2010-11), are you a Highly Qualified Teacher
(HQT) according to your state’s requirements? __ Yes __ No |
Findings
All participants responded “Yes” to this item.
Four people did not express any doubt about their status, and seemed to define HQT in a way that was consistent with the definition provided.
Two teachers, both from Florida, were confident they were HQT based on the fact that they were certified in their subject areas and had been teaching for a long time.
Three people said they were unsure if they were HQT based on this item because they did not know if they had “demonstrated competency” in their subject areas. All three decided they had demonstrated competency based on their belief that they were good teachers or a positive evaluation.
A few participants commented that if a particular state’s standards were higher than the definition given (e.g., if a state required a degree in a specific subject area), some respondents from that state might answer “Yes” to this question even though they were not HQT according to their state’s standards.
Most teachers said the first qualification was clear, accurate, and easy to understand. Two teachers, however, said that it would be more accurate to say that teachers must “have at least a bachelor’s degree.” Two participants thought it should specify that the degree had to be in a specific content area. Three teachers said that in their states, a Master’s degree was required to be HQT.
Most participants felt the second qualification was clear, accurate, and easy to understand. However, one participant was confused because she interpreted the statement to mean that she had to hold some kind of “alternative certification” in addition to full state certification or licensure. However, she still responded “Yes” to the item, even though she knew she did not possess any kind of alternative certification.
Participants were divided over whether all teachers would know their HQT status.
Four participants felt that all teachers responding to this survey would know their HQT status, either because they believed the definition provided was sufficient or that administrators generally communicated this information clearly with their teachers.
The other five participants thought that some teachers might not know their HQT status, particularly new teachers and teachers in private schools. Some thought that if a teacher did not already know what HQT meant, the italicized definition would not be sufficient to explain it to them.
Recommendations
Consider modifying the first qualification to state that “teachers must have at least a bachelor’s degree.”
Provide a clearer explanation of how teachers would “demonstrate competency in the subject area(s) they teach.”
Q4 During this school year (2010-11), will you (or have you): a. Serve as a coach/consultant in a particular subject? __ Yes __ No b. Serve as a mentor coordinator in your school or district? __ Yes __ No c. Serve in a teacher union leadership position? __ Yes __ No d. Give a presentation at a professional conference? __ Yes __ No e. Participate in any other leadership activity at your school or district? __ Yes __ No |
Findings
Four teachers responded “Yes” to (a). Two of these participants defined a coach/consultant as someone who was considered an expert in a content area and gives support to teachers. The other two participants defined a coach/consultant as someone who teaches teachers how to teach or anyone who has something to share with others.
Of the remaining five teachers, three defined a coach/consultant as an expert in a content area and gives support to teachers; one said it would be someone who tutors students; and the remaining participant said it would include an academic, afterschool, or science fair coach.
All but one of the teachers said that they would consider serving as a coach/consultant to be a leadership role; the remaining participant said serving coach or consultant was a resource for other teachers but not necessarily a leader.
Five teachers responded “Yes” to (b). Upon further questioning, it became clear that three of these participants had not served as mentor coordinators. Two of them had been mentors but not mentor coordinators. The other participant said “Yes” because he coordinated a program (although it was unrelated to mentoring) and served as an informal mentor to others.
The remaining participants defined a mentor coordinator as someone who identified potential mentors for new teachers and arranged mentor activities.
All participants considered a mentor coordinator to be a leadership position.
One participant responded “Yes” to (c). A different participant commented that she was not sure if serving as the school’s union representative would count as a “teacher union leadership position” because it was leadership within the school but not necessarily the union.
Four participants responded “Yes” to (d). Most respondents took “professional conference” to mean an education-related meeting outside the school. Two participants, however, believed this could include presenting at a staff meeting; one of these responded “Yes” to the item. All but two of the participants said they considered giving a presentation at a professional conference to be a leadership role. The remaining two participants said it could be considered a leadership role but was not necessarily.
All but one of the participants answered “Yes” to (e). When asked what kinds of activities might be included under (e), participants seemed to consider the phrase “leadership activity” to be very broad. Examples that they suggested included:
Team leader or department head
Facilitating/leading staff development sessions
Working with the principal
Serving as a committee member
Working on the curriculum or developing new courses
Model teaching
Attending a conference
Conducting staff meetings
Serving as a mentor teacher
Participating in the PTA
Organizing events (e.g., Relay for Life)
Coaching sports
Sponsoring student clubs
Taking a group of students on a trip
Recommendations
Consider specifying in (a) that the question is referring to coaches or consultants who work with other teachers, not students.
Consider specifying in (b) exactly what is meant by a mentor coordinator.
If presentations at internal staff meetings should not be included in (d), specify this in the item.
NCES should be aware that when answering (e), respondents are likely to take a very broad view of what is meant by “leadership activity.”
Q5 Were you given any training related to mentoring by your school or district? __ Yes __ No |
Findings
All but one of the participants said this item was clear and easy to understand. The remaining participant said it was unclear whether this question was asking if the respondent had simply been offered mentor training, or if they had participated in the training.
When asked what they would consider to be training, participants mentioned guidance on how to be a mentor, the responsibilities related to mentoring, what to look for in the “mentee” teacher, how to interact with mentees, effective communication, and how to complete the required paperwork.
Participants were asked how they would respond to this question if the only “training” they received was an informal conversation with a more experienced mentor about their role. Half said that they would answer “Yes,” while the other half of participants believed that the question was referring only to more formal training.
Recommendations
Depending on the intent of the question, consider providing clarification as to whether only formal, structured training should be considered when responding to this item.
Q6 To what extent do you feel prepared to be a mentor? __ Not at all prepared __ Somewhat prepared __ Well prepared __ Very well prepared |
Findings
All participants, regardless of whether or not they had received any training to be a mentor or had served as a mentor, said they felt “Very well prepared” or “Well prepared” to be a mentor.
None of the participants said there was anything difficult or unclear about this item. A few pointed out that respondents’ answers would be largely based on their own self-confidence, and would not be an objective measure of preparation.
Recommendations
No modifications are recommended for this item.
Appendix
A:
BTLS Teacher Interview Protocols
Interview
Protocol for National Center for Education Statistics
Teacher
Telephone Interviews
June 2011
Teacher Group: “Movers”
Current teachers who changed schools within the past 5 years (without a break from teaching)
Introduction
“Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF Macro, a research company that the Department of Education has hired to conduct this study. Before we begin, I’d like to ask whether you have copies of the questions that we are going to be talking about today.
The purpose of this interview is to test potential items for the Beginning Teacher Longitudinal Study, a national educational survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any changes to the survey items, the researchers always test them with potential participants to make sure that the items are as easy to understand as possible. In today’s interview, I am going to have you answer some of these items just as you would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the information that the survey writers intended.
As you answer each item I’d like you to ‘think aloud.’ In other words, I would like you to say aloud what you are thinking as you read each question, as you consider the answer choices, and as you finally answer the question. For example, if you are trying to decide which answer to select, please explain why you are unsure. This will help us determine whether the question is truly being understood as it is intended.
Re-Screening
Before you begin, ask the interviewee the question below to confirm that they are qualified for the interview and to obtain background information for the interview:
1) Why did you leave your school and move to a new one? Probe for all reasons that may have contributed to this decision. This information will be used to check their response to Item 1.
Say to the participant: Please go to the first item and answer it just as you would if you were taking the survey. Remember to think out loud as you are going through so I know how you are thinking about it.
Directions for Interviewer:
For each survey item being tested, do the following:
Ask the participant to read the item, consider the potential answer choices, and then select the most appropriate choice—just as if he or she were actually completing this survey. As they do, remind them to “think aloud.” If they are quiet for a period of time, ask them what they are thinking about. When they select an answer, mark it on the answer sheet for that participant.
Allow the participant to answer a complete series (as marked on the protocol) before asking probing questions. Try not to ask any probing or prompting questions as they give their initial answer; if they are having trouble understanding the item or choosing an answer, ask them to describe exactly what they are struggling with.
For each item, record three things:
Their final answer(s)
Notes on any follow-up questions
Any other notes on issues that they brought up with the item, anything they struggled with, or anything else relevant that they said while “thinking aloud.”
1. Indicate the level of importance EACH of the following played in your decision to leave LAST YEAR’S SCHOOL. Mark (X) one box on each line.
I left the position of a pre-K-12 teacher at my last year’s school:
|
Not at all important |
Slightly important |
Somewhat important |
Very important |
Extremely important |
|||||
Personal Life Factors |
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
Assignment and Credential Factors |
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
Salary and Other Job Benefits |
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
Classroom Factors |
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
School Factors |
|
|
|
|
|
|||||
|
Not at all important |
Slightly important |
Somewhat important |
Very important |
Extremely important |
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
Student Performance Factors |
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
Other Factors |
|
|
|
|
|
|||||
|
|
|
|
|
|
|||||
Please specify:
|
|
|||||||||
Testing Goals
Do participants understand what is meant by the additional items?
Think Aloud Notes
Probes
Is there anything about this item that could be confusing or unclear?
How easy was it for you to choose the reasons why you changed schools? How easy was it for you to rate how important these reasons were? Did you have any difficulty answering this question for any reason?
Compare participants’ answers to Item 1 with their open-ended explanation of why they changed schools, and probe on any inconsistencies. For example, if they hadn’t mentioned a reason but then gave it a non-1 answer in Item 1, ask why—or if the first reason they mentioned in their open-ended explanation didn’t get the highest ranking in Item 1, ask why.
Do you think there are any other reasons for changing schools that should be added to this list?
Now I’d like to give you a few reasons teachers might change schools, and ask which response options you think they would belong under. Which item would you select if you changed schools because…
You changed schools because you wanted a teaching job closer to your home?
Part _________ (Part A)You wanted to be able to coach soccer, and your former school didn’t have a soccer team? Part _________ (Part K)
You didn’t like the fact that your pay was determined in part by how your students did on standardized tests? Part _________ (Part R)
You didn’t mind that your pay was determined by how your students performed on standardized tests, but you didn’t like the specific formula that was used to determine the impact of your students’ scores on your pay? Part _________ (Part R)
You moved to another county because your spouse got a new job, and based on where you now lived it made sense for you to change schools? Part _________ (Part A)
Great. Now I would like to ask you what you think is meant by a few of the items.
In your own words, what do you think is meant by Part A? What are some examples of reasons that would fall under Part A?
If the participant did not select Part A for the above reasons, explain: The Census would like people to select Part A for a wide range of reasons, including if they moved because their spouse got another job, or if they changed schools because the new school was closer to where they lived. How can this be communicated more clearly? How could Part A be reworded?
In your own words, what you do think is meant by Part K? What are some examples of reasons for switching schools that might fall under this category?
If the participant did not select Part K for the third reason above, explain: The Census would like people to select Part K for a wide range of reasons, including if they were interested in a specific opportunity at the school, other than teaching. How can this be communicated more clearly?
What do you think is meant by Part R?
Do you think this question is asking whether you were dissatisfied with the specific ways that your compensation was tied to student performance, or whether you were dissatisfied with the fact that your compensation was tied to performance at all, or both?
Say to the participant: Please answer the remaining items just as you would if you were taking the survey. Remember to think out loud as you are going through them so I know how you are thinking about the items.
Testing Goals
Will participants respond to this item or is it too personal?
Think Aloud Notes
Prompts
Is there anything about this item that could be confusing or unclear?
Did you have any difficulty answering this question for any reason?
If you saw this item on a survey, would you be willing to answer it?
Why or why not?
If participant said they would skip it, Is there anything that would make you more likely to respond to this item?
Testing Goals
Do respondents understand that HQT is a specialized certificate?
Will respondents know whether or not they are HQT?
Think Aloud Notes
Prompts
Is there anything about this item that could be confusing or unclear?
Did you have any difficulty answering this question for any reason?
In your own words, what is meant by a Highly Qualified Teacher?
Do you think the definition given for Highly Qualified Teacher is accurate? If not, why not?
Is there any way that the question could be made clearer or easier to understand?
How likely do you think it would be that a teacher responding to this survey wouldn’t know whether or not they were HQT? In what situations do you think they might not know?
Testing Goals
Do participants consider Parts A-E to be examples of “leadership roles”?
What might other leadership activities be?
Think Aloud Notes
Prompts
Is there anything about this item that could be confusing or unclear?
Did you have any difficulty answering this question for any reason?
In your own words, what do you think is meant by Part A?
Let’s go through some of the parts under Item 4.
Part A: Do you consider a coach/consultant to be a leadership role? Why or why not?
Part B: What about mentor coordinator, is that a leadership role? Why or why not?
Part D: Giving a presentation at a professional conference? Why or why not?
Part E: What are some examples of “other leadership activities”? Why or why not?
Testing Goals
What do respondents consider to be training?
Think Aloud Notes
Prompts
Is there anything about this item that could be confusing or unclear?
Did you have any difficulty answering this question for any reason?
What kinds of things would you consider to be “training related to mentoring”?
Imagine you were told that you were going to be a mentor. You met with an experienced mentor who took the time to explain this new role to you. In the context of this question, would you consider that to be “training”?
Testing Goals
None
Think Aloud Notes
Prompts
Is there anything about this item that could be confusing or unclear?
Did you have any difficulty answering this question for any reason?
Interview
Protocol for National Center for Education Statistics
Teacher
Telephone Interviews
July 2011
Teacher Group: “Leavers”
Teachers who left teaching within the past 5 years (may or may not have returned)
Introduction
“Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF Macro, a research company that the Department of Education has hired to conduct this study. Before we begin, I’d like to ask whether you have copies of the questions that we are going to be talking about today.
The purpose of this interview is to test potential items for the Beginning Teacher Longitudinal Study, a national educational survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any changes to the survey items, the researchers always test them with potential participants to make sure that the items are as easy to understand as possible. In today’s interview, I am going to have you answer some of these items just as you would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the information that the survey writers intended.
As you answer each item I’d like you to ‘think aloud.’ In other words, I would like you to say aloud what you are thinking as you read each question, as you consider the answer choices, and as you finally answer the question. For example, if you are trying to decide which answer to select, please explain why you are unsure. This will help us determine whether the question is truly being understood as it is intended.
Re-Screening
Before you begin, ask the interviewee the question below to confirm that they are qualified for the interview and to obtain background information for the interview:
1) Why did you leave teaching? Probe for all reasons that may have contributed to this decision. This information will be used to check their response to Item 1. NOTE: It is more natural to ask about when a teacher started teaching, when they left, and what contributed to that decision.
NOTE: If participant did not leave teaching, or left teaching more than five years ago, he or she does not qualify to be interviewed. Please apologize and say that Joyce will call him or her back.
2) Could you describe to me what you have been doing since you left teaching? OR, if participant has already returned to teaching: Could you describe to me what you did during the period of time you left teaching?
Say to the participant: Please go to the first item and answer it just as you would if you were taking the survey. Remember to think out loud as you are going through so I know how you are thinking about it.
Directions for Interviewer:
For each survey item being tested, do the following:
Ask the participant to read the item, consider the potential answer choices, and then select the most appropriate choice—just as if he or she were actually completing this survey. As they do, remind them to “think aloud.” If they are quiet for a period of time, ask them what they are thinking about. When they select an answer, mark it on the answer sheet for that participant.
Allow the participant to answer a complete series (as marked on the protocol) before asking probing questions. Try not to ask any probing or prompting questions as they give their initial answer; if they are having trouble understanding the item or choosing an answer, ask them to describe exactly what they are struggling with.
For each item, record three things:
Their final answer(s)
Notes on any follow-up questions
Any other notes on issues that they brought up with the item, anything they struggled with, or anything else relevant that they said while “thinking aloud.”
1. Indicate the level of importance EACH of the following played in your decision to leave your pre-K-12 teaching position. Mark (X) one box on each line.
I left the position of a pre-K-12 teacher:
|
Not at all important |
Slightly important |
Somewhat important |
Very important |
Extremely important |
Personal Life Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Assignment and Credential Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Salary and Other Job Benefits |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Career Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Classroom Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
School Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Student Performance Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Other Factors |
|
|
|
|
|
|
|
|
|
|
|
Please specify:
|
|
||||
Testing Goals
Do participants understand the new items?
Think Aloud Notes
Probes
Is there anything about this item that could be confusing or unclear?
Did you have any difficulty answering this question for any reason? How easy was it for you to choose the reasons why you left teaching? How easy was it for you to rate how important these reasons were?
Compare participants’ answers to this item with their earlier open-ended explanation of why they left teaching, and probe on any inconsistencies. For example, if they hadn’t mentioned a reason but then gave it a non-1 answer in Item 1, ask why—or if the first reason they mentioned in their open-ended explanation didn’t get the highest ranking in Item 1, ask why.
Do you think there are any other reasons for leaving teaching that should be added to this list?
Now I’d like to give you a few reasons teachers might leave the field, and ask which response options you think they would belong under. Which item would you select if you left teaching because…
You had a teaching job but decided that you wanted a job closer to your home, and a non-teaching job was the best option in terms of location?
Part _________ (Part A)You didn’t like the fact that your pay was determined in part by how your students did on standardized tests? Part _________ (Part V)
You didn’t mind that your pay was determined by how your students performed on standardized tests, but you didn’t like the specific formula that was used to determine the impact of your students’ scores on your pay? Part _________ (Part V)
You moved to another county because your spouse got a new job, and based on where you now lived it no longer made sense for you to teach? Part _________ (Part A)
Great. Now I would like to ask you what you think is meant by a few of the items. In your own words, what do you think is meant by Part A? What are some examples of reasons that would fall under Part A?
If the participant did not select Part A for the above reasons, explain: The Census would like people to select Part A for a wide range of reasons, including if they moved because their spouse got another job, or if they changed jobs to be closer to where they lived. How can this be communicated more clearly? How could Part A be reworded?
What do you think is meant by taking courses or pursuing a degree to improve career opportunities “WITHIN the field of education,” as in Part K? Can you give an example of a situation that might fall under this category?
What do you think is meant by taking courses or pursuing a degree “OUTSIDE the field of education,” as in Part L? Can you give an example of a situation that might fall under this category?
What is the difference between Part K and Part L?
What do you think is meant by Part V?
Do you think this question is asking whether you were dissatisfied with the specific ways that your compensation was tied to student performance, or whether you were dissatisfied with the fact that your compensation was tied to performance at all, or both?
Say to the participant: Please answer the remaining items just as you would if you were taking the survey. Remember to think out loud as you are going through them so I know how you are thinking about the items.
Note: Questions 2-6 are the same as “Movers” guide.
Interview
Protocol for National Center for Education Statistics
Teacher
Telephone Interviews
July 2011
Teacher Group: “Returners”
Teachers who had a break in service but returned to teaching in the past 5 years
Introduction
“Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF Macro, a research company that the Department of Education has hired to conduct this study. Before we begin, I’d like to ask whether you have copies of the questions that we are going to be talking about today.
The purpose of this interview is to test potential items for the Beginning Teacher Longitudinal Study, a national educational survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any changes to the survey items, the researchers always test them with potential participants to make sure that the items are as easy to understand as possible. In today’s interview, I am going to have you answer some of these items just as you would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the information that the survey writers intended.
As you answer each item I’d like you to ‘think aloud.’ In other words, I would like you to say aloud what you are thinking as you read each question, as you consider the answer choices, and as you finally answer the question. For example, if you are trying to decide which answer to select, please explain why you are unsure. This will help us determine whether the question is truly being understood as it is intended.
Re-Screening
Before you begin, ask the interviewee the question below to confirm that they are qualified for the interview and to obtain background information for the interview:
1) Why did you return to teaching? Probe for all reasons that may have contributed to this decision. This information will be used to check their response to Item 1. NOTE: It is more natural to ask about when a teacher started teaching, left, and returned.
Say to the participant: Please go to the first item and answer it just as you would if you were taking the survey. Remember to think out loud as you are going through so I know how you are thinking about it.
Directions for Interviewer:
For each survey item being tested, do the following:
Ask the participant to read the item, consider the potential answer choices, and then select the most appropriate choice—just as if he or she were actually completing this survey. As they do, remind them to “think aloud.” If they are quiet for a period of time, ask them what they are thinking about. When they select an answer, mark it on the answer sheet for that participant.
Allow the participant to answer a complete series (as marked on the protocol) before asking probing questions. Try not to ask any probing or prompting questions as they give their initial answer; if they are having trouble understanding the item or choosing an answer, ask them to describe exactly what they are struggling with.
For each item, record three things:
Their final answer(s)
Notes on any follow-up questions
Any other notes on issues that they brought up with the item, anything they struggled with, or anything else relevant that they said while “thinking aloud.”
1. Indicate the level of importance EACH of the following played in your decision to return your pre-K-12 teaching position. Mark (X) one box on each line.
I returned to the position of a pre-K-12 teacher:
|
Not at all important |
Slightly important |
Somewhat important |
Very important |
Extremely important |
Personal Life Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Salary and Other Job Benefits |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Career Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Assignment and Credential Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
School Factors |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Please specify:
|
|
|
|
|
|
Testing Goals
Do participants understand the new items?
Think Aloud Notes
Probes
Is there anything about this item that could be confusing or unclear?
How easy was it for you to choose the reasons why you returned to teaching? How easy was it for you to rate how important these reasons were? Did you have any difficulty answering this question for any reason?
Compare participants’ answers to this item with their earlier open-ended explanation of why they returned to teaching, and probe on any inconsistencies. For example, if they hadn’t mentioned a reason but then gave it a non-1 answer in Item 1, ask why—or if the first reason they mentioned in their open-ended explanation didn’t get the highest ranking in Item 1, ask why.
Do you think there are any other reasons for returning to teaching that should be added to this list?
Now I’d like to give you a few reasons teachers might return to teaching, and ask which response options you think they would belong under. Which item would you select if you returned to teaching because…
You had another non-teaching job but decided that you wanted a job closer to your home, and teaching was the best option in terms of location?
Part _________ (Part A)You were laid off from a non-teaching job, so you went back to teaching?
Part _________ (Part C)You had a job, but returned to teaching because it paid more money?
Part _________ (Part C)You liked the fact that your pay would be determined in part by how your students did on standardized tests? Part _________ (Part E)
You liked the specific formula that would be used to determine the impact of your students’ scores on your pay? Part _________ (Part E)
You moved to another county because your spouse got a new job, and based on where you now lived it made sense for you to teach? Part _________ (Part A)
You wanted to be able to coach soccer, so you decided to teach?
Part _________ (Part N)
Great. Now I would like to ask you what you think is meant by a few of the items.
In your own words, what do you think is meant by Part A? What are some examples of reasons that would fall under Part A?
If the participant did not select Part A for the above reasons, explain: The Census would like people to select Part A for a wide range of reasons, including if they moved because their spouse got another job, or if convenience teaching job was the best option because it was close to where they lived. How can this be communicated more clearly? How could Part A be reworded?
In your own words, what you do think is meant by Part C? What are some examples of reasons for returning to teaching that might fall under this category?
What do you think is meant by Part E?
Do you think this question is asking whether you liked the specific ways that your compensation would be tied to student performance, or whether you liked the fact that your compensation would be tied to performance at all, or both?
In your own words, what you do think is meant by Part N? What are some examples of reasons for returning to teaching that might fall under this category?
If the participant did not select Part N for the fifth reason above, explain: The Census would like people to select Part N for a wide range of reasons, including if they were interested in a specific opportunity at the school, other than teaching. How can this be communicated more clearly?
Now, please look at Part O. In your own words, what is this item saying?
What this item was trying to capture was the fact that people might decide to return to teaching because they could retain the benefits or privileges they had accumulated earlier—for example, they could return to the same salary step or could count prior years of teaching towards retirement. Is this what you thought the question meant?
Is there any way the question could be worded more clearly?
Say to the participant: Please answer the remaining items just as you would if you were taking the survey. Remember to think out loud as you are going through them so I know how you are thinking about the items.
Note: Questions 2-6 are the same as “Movers” guide.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Joyce.Clingan |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |
© 2024 OMB.report | Privacy Policy