Cog Lab Report

Cognitive Testing of
Beginning Teacher Longitudinal
Study Survey Items
Summary of Findings and Recommendations

September 2010
Submitted to:
United States Census Bureau

Submitted by:

Macro International Inc.
11785 Beltsville Drive, Suite 300
Calverton, MD 20705
www.macrointernational.com

Cognitive Testing of
Beginning Teacher Longitudinal Study
Survey Items
Summary of Findings and Recommendations

September 2010

Submitted to:
United States Census Bureau

Submitted by:

Macro International Inc.
11785 Beltsville Drive, Suite 300
Calverton, MD 20705
www.macrointernational.com

BACKGROUND
In the spring of 2010, the Census Bureau contracted with ICF Macro, a research and evaluation firm in
Calverton, Maryland, to plan and carry out a series of cognitive interviews with current and former
teachers. The purpose of these interviews was to gather feedback on a number of proposed items for
the surveys that will be a part of the Beginning Teacher Longitudinal Study (BTLS), a national study
conducted by the National Center for Education Statistics (NCES) and the Census Bureau. The BTLS
survey will be administered annually for 10 years to a cohort of teachers in order to gain insight into
teachers’ responses to transitions.
This report summarizes the methodology used in conducting the interviews and feedback ICF Macro
received from participants about the survey items. In addition, this report provides ICF Macro’s
recommendations for revisions to the proposed SASS items.

SUMMARY OF METHODOLOGY
Recruitment Protocol and Summary of Participants
ICF Macro conducted cognitive interviews with 18 current or former teachers. Teachers were recruited
from several sources, including a list provided by NCES of schools in the SASS sampling frame, a list
purchased from an outside vendor, and an ICF Macro database of educators. An email was sent out to
potential participants that described the study and offered an incentive to those who participated.
Interested teachers who contacted ICF Macro by telephone or email were then screened and scheduled
for an interview.
Depending on their circumstances, teachers were classified as “Movers,” “Leavers,” or “Returners,”
and sent the appropriate interview protocol. Movers were defined as teachers who had left one school
and moved to another within the past five years. Leavers were defined as teachers who had left the
teaching profession in the past five years. Returners were defined as teachers who, in the past five
years, had left teaching for at least one year and then returned to teaching. ICF Macro interviewed six
teachers in each group.
Teachers represented each of the school levels (elementary, middle, and high) as well as public, private,
and charter schools. Interviews were conducted with teachers in Illinois, Kentucky, Louisiana,
Maryland, Missouri, New Hampshire, New Jersey, New York, North Carolina, Ohio, Pennsylvania,
Tennessee.

Interview Protocol
Each interview lasted approximately 60 minutes and was conducted by phone. Prior to each interview,
the participants were emailed a copy of the survey items and told to print them out but not to read
them. The interview protocols were different for each group (Movers, Leavers, and Returners). At the

Page 1 of 15

beginning of each interview, the interviewer rescreened the participants to ensure the participant was
eligible for the study and classified in the correct group.
During the interview, participants were asked to answer the proposed BTLS items as they normally
would if they were responding to the survey. As they answered each item, they were also asked to
“think aloud”—that is, to describe out loud their thought process as they read and responded to each
item. After the participant answered a series of items, the interviewer then asked appropriate follow-up
questions or probes. A copy of the original interview protocol is included in this report as Appendix A.
Following the proposed items for the BTLS questionnaire, all participants were asked to read a “Study
Information Form” that would be provided to respondents. They were then asked a series of questions
about the information presented in that form to test their comprehension of the content. The Study
Information Form that participants were shown is included in this report as Appendix B.

SUMMARY OF PARTICIPANT FEEDBACK AND RECOMMENDATIONS
The following section of this report summarizes the results of this study. For each item being tested, we
provide the wording and response options, a list of relevant findings, and our recommendations for
revisions to the item based on those findings.
Q1 (Matrix Item; Movers, Leavers, Returners)
NOTE: The matrix for each group is found in Appendix C.
Movers: Indicate the level of importance EACH of the following played in your decision to leave
LAST YEAR’S school.
Leavers: Indicate the level of importance EACH played in your decision to leave your pre-K–12
teaching position.
Returners: Indicate the level of importance EACH played in your decision to return to the position of a
pre-K-12 teacher.

Findings
•

While some of the participants said this item initially appeared overwhelming, none of the
participants felt it was actually too burdensome to complete. They commented that the way it is
broken up into “factors” makes it easier to complete.

•

Before looking at this item, participants were asked to give their own description of why they left
teaching, returned to teaching, or left their school. In almost all cases, participants’ responses to this
item matched this description. In a few cases, however, there were some discrepancies:


One Leaver said she left teaching to go to graduate school to become a school counselor.
However, she rated “take courses to improve opportunities INSIDE the field of education” as

Page 2 of 15

“not at all important,” because she seemed to consider school counseling as outside the field of
education. She also rated “pursue a position other than K-12 teaching” as “extremely
important,” even though she was not immediately pursuing a position.


Two Returners rated some of the items based upon the level of importance they played in their
decisions to return to teaching, but other items based on the level of importance they played in
their decisions to leave teaching in the first place. For example, one teacher rated “making a
difference in students’ lives” as “very important.” However, she later explained that this was not a
factor in her return to teaching, but rather, had made her decision to leave teaching very difficult.



At least one Returner answered some of the items (particularly those later in the list) in terms of
the extent to which they were true, rather than the extent to which they contributed to his
decision to return to teaching. Initially, he explained that health care was his primary reason for
returning to teaching. However, he marked several unrelated items as “very important”
because he agreed that they were good reasons (e.g., “offered the grade/subject area he
wished,” “liked the school schedule/calendar”). Several other respondents found themselves
beginning to answer the items in this way, but reminded themselves of the intent of the item.

•

Participants were also asked, hypothetically, how they would indicate that their spouse’s relocation
was a factor in their decision, some participants said they would select both Factors A (because I
moved) and B (other personal life reasons), some said they would select only Factor B, and one said
she would select only Factor A.

•

Several participants suggested that other factors should be added to the list as reasons a teacher
would leave a school or leave teaching:

•



Dissatisfaction with colleagues (suggested by two participants)



Too many demands on time outside of the classroom (suggested by three participants)

Returners suggested that other factors should be added to the list as reasons a teacher would return
to teaching:


Because they missed their colleagues,



Opportunities for professional development


Opportunity for promotion (suggested by two participants)

Recommendations
•

Consider revising the question stem to emphasize that responses should be based on the level of
importance the factor played in the decision made, and not in general. For example, “Indicate the
level of importance EACH of the following played in your decision to leave LAST YEAR’S school:”

Page 3 of 15

•

Consider adding the factors “dissatisfaction with colleagues” and “too many demands on time
outside of the classroom” to the Movers and Leavers versions.

•

Consider adding the factor “opportunities for professional development and/or advancement” to
the Returners version.
Q2 (Movers, Leavers, Returners)

During the 2009 calendar year, did you renew your teaching certificate?
__ Yes
__ No

Findings
•

Five of the 18 participants responded “Yes” to this item. However, in four of the five cases it was
not clear that the participant had truly renewed their certificate in 2009:


One respondent said “Yes” to this item because she changed certifications (from initial to
professional) in 2009.



One respondent said “Yes” to this item because she submitted the paperwork to change
certifications (from initial to standard) in 2009. However, she has not yet received her
standard certification.



One respondent renewed his certificate in 2008, but said “Yes” to this item because it was “still
renewed” (i.e., current) in 2009.



One respondent said “Yes” to this item, explaining, “I didn’t do anything to renew it but I say
‘Yes’ because the state renews it automatically.”

•

Three participants suggested adding a “not applicable” response option. Two of these participants
taught in private schools and were not certified. For the third, the state in which she teaches does
not require teachers to renew their teaching certificate.

•

This item was shown to participants after a series of items that referenced the “school year.”
Participants did not seem to have any difficulty going from “school year” to “calendar year.” Two
teachers, however, said they preferred the term “school year.”

Page 4 of 15

Recommendations
•

Depending on the intent of the item, consider revising the question as “Was your teaching
certificate renewed during the 2009 calendar year?”

•

If teachers who are not certified will be responding to this item, consider adding a “not applicable”
response option.
Q3 (Movers, Leavers, Returners)

During the 2009 calendar year, did you add a content area, field, or grade levels to your teaching
certificate? (These additions to existing, valid certifications may be termed endorsements,
authorizations, additional areas, or extensions.)
__ Yes
__ No

Findings
•

None of the participants had any difficulty with this item. All understood the terms used.

•

One participant suggested adding a “not applicable” option for those who are not certified.

Recommendations
•

If teachers who are not certified will be responding to this item, consider adding a “not applicable”
response option.
Q4 (Movers, Leavers, Returners)

During the 2009 calendar year, did you complete a master’s degree, educational specialist degree or
professional diploma, doctoral degree (PhD, EdD), or professional degree (MD, JD, DDS)?
__ Yes
__ No

Findings
•

No participants had any difficulty responding to this item.

Recommendations
•

No modifications are recommended for this item.

Page 5 of 15

Q5 (Movers, Leavers, Returners)
After the 2008-09 school year, have you enrolled in any courses (degree program, certificate, or nondegree) at a college or university?
__ Yes
__ No

Findings
•

All participants were able to report for this time frame. Several were initially confused by the
phrase “after the 2008-09 school year,” but this did not appear to affect their responses.

•

When asked how they would respond to this item if they had enrolled in a course in July of 2009, all
participants said they would respond “Yes.”

•

All participants who responded “Yes” to this item were referring to courses taken as part of a
degree program.

•

Although it was not relevant to the time frame in question, one participant wondered if she would
count CEU credits she earned at a conference that were later validated by a university. She decided
that she would probably not count them, but felt the item was unclear on this issue.

Recommendations
•

To help respondents understand which courses to include, consider providing an instruction on the
kinds of courses that should not be counted in this item.
Q6 (Movers, Leavers, Returners)

Including yourself, how many family members were living in your household or were financially
dependent on you (or your spouse) during 2010?

Findings
•

Most participants commented that they answered this item in a similar way to how they would on
their tax return.

•

Of the seventeen participants who responded to this item, two did not include themselves in the
count. One of the two responded “0”; the other responded “2.”

Page 6 of 15

•

When asked whether they would count a roommate in their response to this item, all but two said
they would not. The other two said they would consider a roommate to be part of their “household.”

•

When asked whether they would count a financially dependent child who lived in another state in
their response to this item, all but one said they would. The remaining participant indicated that
she would not count the child because he or she would not be part of her “household.”

•

When asked whether they would count a romantic partner who they lived with but were not
married to, participants were almost evenly split. For those who would, the most common reason
was that they would consider the romantic partner as part of the family. Those who would not
count a romantic partner typically said it was because they would not count them as part of the
family, or because they would not consider them to be financially dependent.

Recommendations
•

Consider including an instruction as to whether to count romantic partners in the response to
this item.

•

Consider adding an instruction to respondents to use the number of dependents they report on
their tax return as a guide.
Q7 (Movers, Leavers, Returners)

How many family members counted above were under 5 years of age?

Findings
•

Several participants mentioned that they were not sure if children who were exactly 5 years old
should be included in this item. One participant had a 5 year old child and included this child in
her count.

•

A few participants said this item was initially confusing to them because both the words “above” and
“under” were in the same sentence. They had to reread the item in order to understand the item.

Recommendations
•

Consider rewording the item as follows, “Of the family members included in your response to
Item XX, how many were 4 years of age or younger?”

Page 7 of 15

Q8 (Movers, Leavers)
(Movers) Did you change schools involuntarily (e.g., contract not renewed, laid off, school closed
or merged)?
(Leavers) Did you leave last year’s school involuntarily (e.g., contract not renewed, laid off, school
closed or merged)?
*Report on the most recent time you left pre-K–12 teaching if you’ve left teaching more than once.
__ Yes
__ No  go to item 5

Findings
•

Participants did not have any difficulties with this item.

•

Three participants indicated they had changed schools or left schools involuntarily. They gave the
following reasons: funding issues, budget cuts, and contract not renewed following medical leave.

Recommendations
•

No modifications are recommended for this item.
Q9 (Movers, Leavers)

(Movers) Which of the following best describes the reason why you changed schools involuntarily?
(Leavers) Which of the following best describes the reason why you involuntarily left the school?
__ Budget cuts or budget shortfalls
__ Reduced pupil enrollment
__ School and/or district merger or school closed
__ Transfer required by school or district (Note: Movers only)
__ I did not meet Highly Qualified Teacher (HQT) requirements
(Generally, to be Highly Qualified, teachers must 1) have a bachelor’s degree; 2) hold full state
certification or licensure, including an "alternative certification"; and 3) demonstrate competency
in the subject area(s) they teach. The HQT requirement is a provision under No Child Left
Behind [NCLB].)
__ My contract was not renewed for unknown reasons
__ My contract was not renewed for other reason(s)
( go to item 6)

Page 8 of 15

Findings
•

Participants did not have any difficulties with this item.

•

The participants who said “Yes” to the previous item selected the appropriate reason in this item.

•

Participants agreed that tenured teachers would always know why they were forced to leave a
school involuntarily. Some participants felt that it was possible, though highly unlikely, that new
teachers or teachers working for an “at will” employer might not know why they were forced to
leave. A few said it was more likely that teachers would know why they were forced to leave but
would be unwilling to share the reason.

•

Two participants suggested adding a space to specify why their contract had not been renewed for
“other reason(s).”

•

One participant suggested adding “student performance” as a response option, as it might apply to
teachers in private or charter schools.

Recommendations
•

Consider adding a space to specify “other reason(s)” why the contract was not renewed.
Q10 (Movers, Returners)

Does your current school offer tenure?
__ Yes
__ No  go to item _

Findings
•

Most of the participants did not have any difficulty with this item.

•

Two participants found it somewhat confusing because tenure is offered through their district and
not through their school. They both responded “Yes” to this item.

Recommendations
•

Consider revising the item as follows, “Does your current school, school system, or state
offer tenure?”

Page 9 of 15

Q11 (Movers, Returners)
Are you tenured at your current school?
__ Yes
__ No

Findings
•

Most of the participants did not have any difficulty with this item.

Recommendations
•

No modifications are recommended for this item.
Q12 (Movers, Returners)

Do any of the following people or groups conduct your formal evaluation at this school?
a. Principal
__ Yes
__ No
b. Vice principal or assistant principal
__ Yes
__ No
c. Your peers
__ Yes
__ No
d. Outside group (e.g., consultant)
__ Yes
__ No

Findings
•

Most participants did not have difficulty with this item.

•

Participants defined “formal evaluation” in a similar way—that is, that they are observed by an
administrator who completes an official report and gives them feedback. However, some participants
expressed confusion over what this item was asking. This led to some false positive responses.


One teacher selected “Yes” to principal, vice principal, and outside group—however, only the
vice principal was actually responsible for conducting evaluations at his school. At his school,

Page 10 of 15

the principal conducts informal observations, and state-level personnel come into the school to
provide needs-based training to staff because they are a “school in warning.”


•

•

Another teacher selected “Yes” for all four parts of this item, even though the principal and vice
principal were the only people who conduct formal evaluations. In Part (c) she counted a
former teacher (i.e., a peer) who is now employed by the district to conduct informal
evaluations and coach teachers. In Part (d), she was referring to accreditation evaluators.

Two participants were not sure how to respond because their schools do not have a principal or
vice principal.


For one participant, who was an art teacher, the art director or academic director conducts the
formal evaluations. This participant selected “No” for all parts of this item.



In one non-traditional, performing arts school, the school president and outside consultants
conduct formal evaluations. This participant respond “Yes” to Part (d) and “No” to the other
parts of this item.

Evaluators that participants considered under Part (d) (“outside group”) included: district-level
staff, state-level staff, school board staff, accreditation evaluators, and consultants hired by the
school or district.

Recommendations
•

Consider modifying the item as follows, “Do any of the following people or groups conduct your
FORMAL EVALUATION at this school?”

•

Consider including an instruction indicating that respondents should not consider informal
evaluations in their answer.

•

Consider adding part (e): “Other (specify)” so that teachers who work in non-traditional schools
can provide information on who conducts their formal evaluations.

•

Depending on the intent of the item, consider adding “district-level staff” as an example in part (d),
or as a separate category.

Page 11 of 15

Q13 (Movers)
Do you CURRENTLY TEACH any regularly scheduled class(es) in any of grades pre-K–12?
* If you work as a library media specialist or librarian at your current school, do not include classes in
which you teach students how to use the library (e.g., library skills or library research).
* If you teach a particular specialty either within or outside of a regular classroom (e.g., reading specialist,
special education teacher, English as a Second Language teacher), include that time as a regularly
scheduled class.
__ Yes
__ No

Findings
•

Based on the background information collected at the beginning of the interview, all participants
responded accurately to this item.

•

Participants were asked how they thought a special education teacher who did not teach her own
class but went into different classes to help certain students would answer this item.


Three participants said this teacher should respond “Yes” and gave the following reasons:
because the students she worked with would be her “class”; because the directions say to count
a special education teacher; and because the teacher would be “regularly scheduled” in
those classrooms.



Two teachers said this teacher should respond “No” because she would not be the head teacher
or because she would not teach regularly scheduled classes.



The remaining participant said the teacher should respond “Yes” if the students were officially
assigned to her, but “No” if they were not.

Recommendations
•

Based on these findings, there may be some inconsistency in how special education teachers choose
to respond to this item. However, because there is already an instruction addressing this it is not
clear how the item could be improved in this respect.
Q14 (Movers)

Were you teaching in the same school during 2009–2010 as you were during the 2008–2009 school year?
__ Yes  go to item 6
__ No

Page 12 of 15

Findings
•

Participants did not have any difficulties with this item.

Recommendations
•

No modifications are recommended for this item.
Q15 (Leavers)

What is your current MAIN occupational status?
__ Working for a school or school district in a position in the field of K–12 education, but not as
a regular K-12 classroom teacher
__ Working in a position in the field of pre-K or postsecondary education  go to item 3
__ Working in an occupation outside the field of education, including military service  go to
item 3
__ Student at a college or university  go to item 3
__ Caring for family members  go to item 3
__ Retired  go to item 3
__ Disabled  go to item 3
__ Unemployed  go to item 3
__ Other  go to item 3

Findings
•

Most participants did not have any difficulty with this item.

•

One former teacher who worked for an after school mathematics and reading franchise was not
sure which option to select. She felt the first option did not apply because she does not work for a
school or school district. The second option did not apply because she does not work in pre-K or
postsecondary. The third option did not apply because she works in the field of education.
Ultimately, she selected “Other.”

•

When asked what the phrase “working in the field of K-12 education” meant, all participants said
working in a school but not necessarily as a teacher. Some participants also mentioned working for
the district or working directly with K-12 children.

Recommendations
•

No modifications are recommended for this item.

Page 13 of 15

Q16 (Leavers)
Is your current main occupation a –
*If you have more than one position, mark (X) the position for which you spend the most time.
__ Principal/school head/dean
__ Assistant principal
__ School district administrator
__ Librarian/Library technician
__ Instructional coordinator
__ Academic coach/specialist
__ Teacher assistant/aide
__ Counselor or school psychologist
__ Short-term substitute
__ Other occupation

Findings
•

Two teachers were confused by the “academic coach/specialist” response option.


One teacher had difficulty because her current position is as an “instructional coach.” She had
trouble deciding whether that position was best represented by “instructional coordinator” or
“academic coach/specialist.” Ultimately she selected “instructional coordinator.” She explained
that she interpreted the coordinator option as someone who gives support/instruction to
teachers in any content area, while the coach would give support in a specific content area.



Another teacher was not sure what “specialist” meant. She suggested that it would be an art or
music teacher (i.e., a teacher who teaches “specials”). She did not think art or music teachers
would consider themselves regular classroom teachers.

Recommendations
•

No modifications are recommended for this item.
Study Information Form
NOTE: The Study Information Form that was tested is provided as Appendix B.

Page 14 of 15

Findings
•

None of the participants expressed any confusion about anything they read in the Study
Information Form.

•

When asked whether there was any other information they would want to know about the study
that wasn’t provided on the form, participants mentioned the following:

•



Who will be using the data?



What kinds of reports will be created, and who will read them?



How will the results of the survey help me and my school?

After reading the Form, all 18 participants understood:


The purpose of the study;



That the study is voluntary;



How they could notify the Census Bureau that they did not wish to participate;



How long it takes on average to complete the survey; and



How the response data will be used.1

•

All but one participant understood how the confidentiality of their data would be ensured; the
remaining participant mistakenly thought that no personal information would be collected.

•

While all participants understood that both NCES and the Census Bureau were responsible for the
survey, a few were unsure about their respective roles.

Recommendations
•

The current content of the form is clear to readers; no revisions are necessary.

Consider providing more specificity about how the data will be used (e.g., “by policymakers to inform
decisions that affect teachers and schools”).

1 While all participants understood the information on the form about how the data will be used, some thought that more specific
information should be provided.

Page 15 of 15

APPENDIX A
BTLS TEACHER INTERVIEW PROTOCOL

Interview Protocol for Beginning Teacher
Longitudinal Survey
June 2010
Introduction
“Thank you for agreeing to assist us with this project. My name is _____________, and I work for ICF Macro, a research
company that the Department of Education has hired to conduct this study. Before we begin, I’d like to ask whether you
have copies of the questions that we are going to be talking about today.
The purpose of this interview is to test potential items for the Beginning Teacher Longitudinal Study, a national educational
survey that is conducted by the National Center for Education Statistics and the Census Bureau. Before they make any
changes to the survey items, the researchers always test them with potential participants to make sure that the items are
as easy to understand as possible. In today’s interview, I am going to have you answer some of these items just as you
would if you were really taking the survey so that we can make sure that they are clear and that they are soliciting the
information that the survey writers intended.
As you answer each item I’d like you to ‘think aloud.’ In other words, I would like you to say aloud what you are thinking as
you read each question, as you consider the answer choices, and as you finally answer the question. For example, if you are
trying to decide which answer to select, please explain why you are unsure. This will help us determine whether the
question is truly being understood as it is intended.

Re-Screening
Before you begin, ask the interviewee the screening questions below for their group just to confirm
that they are qualified for the interview:
(Movers) Why did you leave your school and move to a new one?
• Probe for all reasons that may have contributed to this decision.
• (This information will be used to check their response to Item 5.)
(Leavers) Why did you leave teaching?
• Probe for all reasons that may have contributed to this decision.
• Could you describe to me what you have been doing since you left teaching?
• OR, if participant has already returned to teaching: Could you describe to me what you did during the period of
time you left teaching?
(Returners) Why did you return to teaching?
• Probe for all reasons that may have contributed to this decision.
• (This information will be used to check their response to Item 1.) NOTE: it is more natural to ask about when/why
participant started teaching, left, and returned.

Page A-1

Directions for Interviewer:
For each item, do the following:
a)

Ask the participant to read the item, consider the potential answer choices, and then select the most appropriate
choice—just as if he or she were actually completing this survey. As they do, remind them to “think aloud.” If they
are quiet for a period of time, ask them what they are thinking about. When they select an answer, mark it on the
answer sheet for that participant.

b) Allow the participant to answer a complete series (as marked on the protocol) before asking probing questions. Try
not to ask any probing or prompting questions as they give their initial answer; if they are having trouble
understanding the item or choosing an answer, ask them to describe exactly what they are struggling with.
c)

For each item, record three things:
a. Their final answer(s)
b. Notes on any follow-up questions
c. Any other notes on issues that they brought up with the item, anything they struggled with, or anything
else relevant that they said while “thinking aloud.”

Questionnaire Items
Q1: Matrix Item
(Movers) Indicate the level of importance EACH of the following played in your decision to leave LAST YEAR’S school.
(Leavers) Indicate the level of importance EACH played in your decision to leave your pre-K-12 teaching position.
(Returners) Indicate the level of importance EACH played in your decision to return to the position of a pre-K-12 teacher.
•
•
•

Is there anything about this item that could be confusing or unclear?
Did you find answering this question to be at all overwhelming, due to the length of the list?
Let’s take a look at the items listed under each of the categories. For each group, ask: Are any items redundant?
Too narrow? Too broad?
o Personal life factors
o Assignment and credential factors
o Salary and other job benefits
o Career factors
o Classroom factors
o School factors
o Student performance factors
o Other factors

•

(Movers) How easy was it for you to choose the reasons why you changed schools?
o How easy was it for you to rate how important these reasons were?
o Did you have any difficulty answering this question for any reason?

•

(Leavers) How easy was it for you to choose the reasons why you left teaching?
o How easy was it for you to rate how important these reasons were?
o Did you have any difficulty answering this question for any reason?

•

(Returners) How easy was it for you to choose the reasons why you returned to teaching?
o How easy was it for you to rate how important these reasons were?
o Did you have any difficulty answering this question for any reason?

Page A-2

•
•
•

(Movers) Do you think there are any other reasons for switching schools that should be added to this list?
(Leavers) Do you think there are any other reasons for leaving teaching that should be added to this list?
(Returners) Do you think there are any other reasons for returning to teaching that should be added to this list?

•

(Movers) Imagine that you switched schools because your spouse accepted a new job and you had to move. How
would you answer this question in that situation?

Q2 (Movers, Leavers, Returners) During the 2009 calendar year, did you renew your teaching certificate?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
o If yes, on what date did you renew your teaching certificate? (If date reported is not in 2009, ask why.)
o If no, when was the last time you renewed your teaching certificate? When is the next time you’ll need to
renew your teaching certificate? How often do you need to renew your certificate?
Q3 (Movers, Leavers, Returners) During the 2009 calendar year, did you add a content area, field, or grade levels to your
teaching certificate?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• Do you know what is meant by “adding a content area, field, or grade levels to your teaching certificate”? What
term is used in your school to describe this?
• Does the sentence in parentheses make sense? Which of these terms have you heard before? Do you think that all
of these terms mean what this item says they mean?
Q4 (Movers, Leavers, Returners) During the 2009 calendar year, did you complete a master’s degree, educational
specialist degree or professional diploma, doctoral degree (Ph.D., Ed.D.), or professional degree (M.D., J.D., D.D.S.)?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• If yes, on what date did you complete your degree? What type of degree did you complete?
Q5 (Movers, Leavers, Returners) After the 2008-2009 school year, have you enrolled in any courses (degree program,
certificate, or non-degree) at a college or university?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• If yes, how many courses did you take?
o For each of the courses, ask: Please describe the course.
• Was the course(s) part of a degree program, certificate, or non-degree program?
• When did the course begin?
• Did you complete the course? If so, when did the course end? (Figure out how teachers are
interpreting “after the 2008-2009 school year.”)
Q6 (Movers, Leavers, Returners) Including yourself, how many family members were living in your household or were
financially dependent on your (or your spouse) during 2010?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• Who did you include in your number? (Family members, roommates, etc.?) Did they include themselves, as
instructions state?
o If children are included: What are the ages of the children?
• Imagine that you were not married and had no children, and lived with a roommate. Would you include the
roommate in your response to this question? [Note: if participant asks whether they are supporting the roommate
financially, answer “no.”]
o If no: Why not?

Page A-3

•
•

Imagine that you had a child that you supported financially, but who lived in a different state. Would you include
them in your response to this question?
o If no: Why not?
Imagine that you lived with your romantic partner/boyfriend/girlfriend. Would you include them in your response
to this question?
o If no: Why not?

Q7 (Movers, Leavers, Returners) How many family members counted above were under 5 years of age?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
Q8
(Movers) Did you change schools involuntarily (e.g., contract not renewed, laid off, school closed or merged)?
(Leavers) Did you leave last year’s school involuntarily (e.g., contract not renewed, laid off, school closed or merged)?
*Report on the most recent time you left pre-K—12 teaching if you’ve left teaching more than once.
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• If yes, please describe the circumstances around what happened when you changed schools involuntarily.
(Interviewer: Try to figure out if they were fired, forced to transfer, etc.)
• Do the examples in parentheses make sense? Are there other examples that could be added to help clarify the
question?
Q9
(Movers) Which of the following best describes the reason why you changed schools involuntarily?
(Leavers) Which of the following best describes the reason why you involuntarily left the school?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• Do any of the response options overlap? Do any seem too similar or redundant?
• Are there any other response options that should be added to this list?
• How likely do you think it would be that a teacher would not know the reason they were laid off?
Q10 (Movers and Returners) Does your current school offer tenure?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• How would you define “tenure?”
• Does your district have a tenure system?
Q11 (Movers and Returners) Are you tenured at your current school?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
Q12 (Movers and Returners) Do any of the following people or groups conduct your formal evaluation at this school?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• Explain what you think is meant by the term “formal evaluation.”
• Have you been formally evaluated? If so, please describe the evaluation process at your school.
o Who does the evaluations? Does someone from the district come to your school to evaluate you or is it
someone from your school? Is it someone else?
o How often are you evaluated?
• Who did you think of when you saw option (d) “Outside group (e.g. consultant)?”
• Are there any other people (internal and external) that conduct formal evaluations that are not on this list?

Page A-4

Q13 (Movers) Do you CURRENTLY TEACH any regular scheduled class(es) in any of grades pre-K - 12?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• Please describe your role or position in your school. (Interviewer: Check for contradictions by making sure that
their position matches their response to Item 1. If you notice a contradiction, ask why.)
• If you were a special education teacher who didn’t teach your own class, but instead went into different classes to
help certain students, how would you answer this question?
Q14 (Movers) Were you teaching in the same school during 2009-2010 as you were during the 2008-2009 school year?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
Q15 (Leavers) What is your current MAIN occupational status?
• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• What do you think the phrase “working in the field of K-12 education” means?
• Are there any additional answer choices that should appear in this list, but do not?
Q16 (Leavers) Is your current main occupation a -• Is there anything about this item that could be confusing or unclear?
• Did you have any difficulty answering this question for any reason?
• This question will be answered by people who say they are working in the field of K-12 education, but not as a
“regular classroom teacher.” With that in mind, are there any additional answer choices that should appear in this
list, but do not?
Study Information Form (Movers, Leavers, Returners)
• Was there anything in the text that you found difficult to understand?
• Is there information that you would like to know about the study that is not presented in this statement?
• If you wanted to get more information about this study, how would you go about doing that?
• After reading this document, what do you think is the purpose of this study?
o Did participant mention that study will be tracking a set cohort of teachers?
o Did participant mention that data will be collected over an extended period of time?
o Did participant mention that information will be used to study teachers’ transitions, or impact that life
events have on their careers?
• Which government agency is sponsoring the study?
• Which government agency is collecting data from participants?
• Is the study voluntary? How do you know? Note whether the participant sees from the form that it is voluntary,
and WHERE on the form they see that.
• If someone doesn’t want to participate, how can he/she remove himself/herself from the study?
• How long does it take to complete the survey on average?
• How will the data be used?
o Did participant mention that his/her data will be combined with data from other respondents?
o Did respondent mention that the data will be used to create statistical reports?
• What is your understanding of data confidentiality?
o Did participant mention that his/her individual data will never be released?
o Did participant mention that his/her data will be stored on an NCES server?

Page A-5

APPENDIX B
STUDY INFORMATION FORM

Thank you for being a part of the Beginning Teacher
Longitudinal Study!
The U.S. Census Bureau is conducting this survey for the National Center for Education Statistics (NCES) of the
U.S. Department of Education. The purpose of the study is to collect data from a cohort of teachers over time to
get a better understanding of how teachers respond to transitions in their lives. Please know that your
participation in this study is completely voluntary. If you would like to be removed from the study, please
contact Freddie Cross in writing at [email protected]. However, we hope you will want to be part of this
important and exciting study.
How will the data be collected?
The data will be collected through a web-based questionnaire every year for about a decade, beginning in the
2007-08 school year. At the beginning of each data collection, you will receive e-mails from NCES containing a
username, password, and link to the questionnaire. We estimate it will take an average of 20 minutes for you to
complete the survey each year, including the time spent reviewing instructions, searching existing data sources,
gathering the data needed, and completing and reviewing the questionnaire.
How will my information be reported?
The information you provide will be combined with data from other respondents to develop statistical reports
about teachers and schools across the country. No individually-identifiable data will be included in these reports.
Will my data be kept confidential?
All responses are protected by Public Law 107-279, the Education Sciences Reform Act of 2002, Title I, Part E,
Section 183. Your answers will be used for statistical purposes only and will not be disclosed, or used, in
identifiable form for any other purpose except as provided by law. Your responses to the survey are stored on a
secure server at the National Center for Education Statistics.
Where can I get more information about this study?
For more information, go to http://nces.ed.gov/surveys/btls.
This survey is authorized by Title I, Part E, Sections 151(b) and 153(a) of Public Law 107-279, the Education
Sciences Reform Act of 2002. According to the Paperwork Reduction Act of 1995, no persons are required to
respond to a collection of information unless such collection displays a valid OMB control number. The valid
OMB control number for this information collection is 1850-0868.

Page B-1

APPENDIX C
MATRIX ITEM FOR MOVERS, LEAVERS,
AND RETURNERS

ITEM 1 (Movers)
Indicate the level of importance EACH of the following played in your decision to leave LAST YEAR’S SCHOOL:
Not at all
important

Slightly
important

Somewhat
important

Very
important

Extremely
important

Personal Life Factors
A.
B.

Because I moved or wanted to take a job more conveniently
located.
Because of other personal life reasons (e.g., health,
pregnancy/childcare, caring for family).

Assignment and Credential Factors
C.
D.

Because I have not taken or could not pass the required test(s).
Because I was dissatisfied with my job description or
assignment (e.g., responsibilities, grade level, or subject area).

E.

Because I wanted or needed a higher salary.

F.

Because I needed better benefits than I received at last year’s
school.
Because I was concerned about my job security at last year’s
school.

Salary and Other Job Benefits

G.

Classroom Factors
H.
I.
J.

Because I did not have enough autonomy over my classroom at
last year’s school.
Because I was dissatisfied with the large number of students I
taught at last year’s school.
Because I felt that there were too many intrusions on my
teaching time (i.e., time spent with students) at last year’s
school.

School Factors
K.
L.

Because I wanted to teach at my current school.
Because I was dissatisfied with opportunities for professional
development at last year’s school.

M.

Because I was dissatisfied with workplace conditions (e.g.,
facilities, classroom resources, school safety) at last year’s
school.
Because student discipline problems were an issue at last year’s
school.
Because I was dissatisfied with the administration at last year's
school.
Because I was dissatisfied with the lack of influence I had over
school policies and practices at last year’s school.

N.
O.
P.

Student Performance Factors
Q.

R.

S.

Because I was dissatisfied with how student
assessments/school accountability measures impacted my
teaching or the curriculum at last year's school.
Because I was dissatisfied with having some of my
compensation, benefits, or rewards tied to the performance of
my students at last year’s school.
Because I was dissatisfied with the support I received for
preparing my students for student assessments at last year’s
school.

Other Factors
T.

Because of other factors not included in the previous items.
 Please specify:

Page C-1

ITEM 1 (Leavers)
Indicate the level of importance EACH played in your decision to leave your pre-K-12 teaching position:
Not at all
important
A.
B.
C.
D.
E.

F.
G.
H.

I.
J.
K.
L.
M.
N.
O.
P.

Q.
R.
S.
T.

U.

V.

W.

X.

Slightly
important

Somewhat
important

Very
important

Extremely
important

Personal Life Factors
Because I moved or wanted to take a job more conveniently
located.
Because of other personal life reasons (e.g., health,
pregnancy/childcare, caring for family).
Because I decided to retire.
Assignment and Credential Factors
Because I have not taken or could not pass the required test(s).
Because I was dissatisfied with my job description or assignment
(e.g., responsibilities, grade level, or subject area).
Salary and Other Job Benefits
Because I wanted or needed a higher salary.
Because I needed better benefits than I received at last year’s
school.
Because I was concerned about my job security at last year’s
school.
Career Factors
Because I decided to pursue a position other than that of a pre-K12 teacher.
Because I was dissatisfied with opportunities for professional
development at last year's school.
Because I decided to take courses to improve career
opportunities WITHIN the field of education.
Because I decided to take courses to improve career
opportunities OUTSIDE the field of education.
Because I was dissatisfied with teaching as a career.
Classroom Factors
Because I did not have enough autonomy over my classroom at
last year’s school.
Because I was dissatisfied with the large number of students I
taught at last year’s school.
Because I felt that there were too many intrusions on my
teaching time (i.e., time spent with students) at last year’s school.
School Factors
Because I was dissatisfied with workplace conditions (e.g.,
facilities, classroom resources, school safety) at last year’s school.
Because student discipline problems were an issue at last year’s
school.
Because I was dissatisfied with the administration at last year's
school.
Because I was dissatisfied with the lack of influence I had over
school policies and practices at last year’s school.
Student Performance Factors
Because I was dissatisfied with how student assessments/school
accountability measures impacted my teaching or the curriculum
at last year's school.
Because I was dissatisfied with having some of my compensation,
benefits, or rewards tied to the performance of my students at
last year’s school.
Because I was dissatisfied with the support I received for
preparing my students for student assessments at last year’s
school.
Other Factors
Because of other factors not included in the previous items.

 Please specify:

Page C-2

ITEM 5 (Returners)
Indicate the level of importance each of the following played in your decision to return to the position of a pre-K–12
teacher.
Not at all
important
A.
B.

C.
D.
E.

F.
G.
H.
I.
J.
K.

L.
M.

N.
O.
P.

Q.
R.

Slightly
important

Somewhat
important

Very
important

Extremely
important

Personal Life Factors
Because I moved or wanted to take a
job more conveniently located.
Because of other personal life reasons
(e.g., maternity leave ended, change in
childcare or healthcare needs).
Salary and Other Job Benefits
Because I wanted or needed a higher
salary.
Because I was offered a financial
incentive to teach.
Because some of my compensation,
benefits, or rewards are tied to the
performance of my students.
Because I needed the health benefits.
Because I wanted the retirement
package.
Because I wanted job security.
Career Factors
Because I realized I preferred pre-K-12
teaching as a career.
Because I completed the coursework I
was pursuing.
Because I missed being able to make a
difference in the lives of students.
Assignment and Credential Factors
Because I passed the required test(s).
Because I was offered the grade level
or subject area that I wished to teach.
School Factors
Because I wanted to teach at my
current school.
Because I was able to maintain
privileges based on my seniority.
Because I liked the school
schedule/calendar.
Other Factors
Because of other factors not included
in the previous items.
Please specify:

Page C-3