TALIS 2013 Main Study Part B and C

OECD Teaching and Learning International Survey
(TALIS) 2013 Main Study

Supporting Statement Part B

National Center for Education Statistics

Institute Of Education Sciences

U.S. Department Of Education

Washington, D.C.

September 7, 2012

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe

TALIS 2013 will acquire information from classroom teachers of ISCED Level 2 programs and the principals of their schools. ISCED Level 2 is also known as lower secondary education and usually lasts between 2 and 6 years, and begins around age 11. Middle school and junior high (grades 7 through 9) in the United States are classified as ISCED Level 2. The universe for the selection of schools is all schools with at least one ISCED Level 2 program in all states of the United States and the District of Columbia. The outlying insular areas and Department of Defense schools are excluded. Teachers who teach in special needs-only schools, who teach exclusively to adults, who serve as occasional or emergency teachers, or who are on long term leave are not within the scope of the study. However, special education teachers working in regular public and private schools are eligible to participate. Additionally, very small schools and schools in remote areas are excluded from the study. For the main study, 201 schools were selected to participate and within each school a sample of up to 22 ISCED Level 2 teachers will be selected (depending on the number of ISCED Level 2 faculty in each school), along with the school administrator, to be surveyed. In schools with a small number of faculty (i.e., 22 or fewer teachers instructing at grades 7, 8, or 9), all teachers who teach at the target grade levels will be asked to participate. In each sampled school with a large number of faculty (i.e., more than 22 teachers instructing at grades 7, 8, or 9), 22 teachers currently teaching any course in the target grades will be randomly selected to participate. In order to make the sampling more cost-effective, the schools with three or less ISCED Level 2 teachers are excluded from the sampling frame for both public and private schools. The overall percentage of ISCED Level 2 teachers in the excluded smaller schools was 3.1%. This sampling plan was reviewed and approved by the international sampling referee for the TALIS study, Statistics Canada.

B.2 Statistical Methodology

The Technical Standards for the TALIS 2013 main study established by the international governing board include the following:

• The teacher sample size must be a minimum of 3,400 surveyed ISCED Level 2 teachers for the main study, or the National Defined Target Population.

• The school sample size must be a minimum of 200 schools for the main study, or all schools that have ISCED Level 2 teachers in the National Defined Target Population.

• The target cluster size is typically 20 TALIS eligible teachers for the main study, although schools with larger faculties may be asked to contribute up to 22 ISCED Level 2 teachers to the study.

• School response rates must be above 75 percent of sampled schools. If a response rate is below 75 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. Although replacement schools could be called upon as substitutes for non-responding schools, national project managers (NPMs) will be encouraged to do all they can to obtain the participation of the schools in the original sample. As the number of replacement schools increases, the sample loses its probabilistic features and becomes increasingly “purposive.” This can undermine the reliability, validity, and interpretability of the country’s results. Responding schools that yield at least 50 percent of sampled teachers will be considered as participating schools; schools that fail to meet that threshold will be considered as “non participating” even though the number of responding teachers may be enough to contribute to some of the analyses. Countries that experience less than 75 percent sample school participation after replacement have to demonstrate convincingly that their sample is not significantly biased. (For a description of the steps required by NCES Standards for surveys that do not meet established participation rates, see below) TALIS establishes three response rate zones—good, fair, or poor. “Good” means the country’s data will be included in the international sample. “Fair” means that the country’s data is a candidate for not being reported in international comparisons because the participation rate after replacement was less than 75 percent. “Poor” means that the country’s data will not be included in the international comparisons. The TALIS Board of Participating Countries makes the final decision on whether to include the country’s data in international comparisons while taking into account various other factors.

• The overall teacher response rates must be at least 75 percent of sampled teachers.

Overview for TALIS 2013 main study

This study will be based on a stratified two-stage probability sample design. At the first stage the primary sampling units (PSUs) will be individual ISCED Level 2 schools, to be selected systematically with probability proportional to size (PPS) from the stratified sampling frame. At the second stage the secondary sampling units (SSUs) will be the in-scope teachers, selected randomly within the sample schools.

The universe of interest is comprised of schools where ISCED Level 2 education is provided along with the affiliated principals and teachers. According to the Indicators of Education Systems (INES) data collection concept, “the formal definition of a classroom teacher is a person whose professional activity involves the planning, organizing, and conduction of group activities whereby students’ knowledge, skills, and attitudes develop as stipulated by educational programs. In short, it is one whose main activity is teaching.”

Target Populations

TALIS was designed to cover all ISCED Level 2 teachers in a participating country. TALIS identified policy issues that encompass the classroom, teacher, the school, and school leadership. No subject matter was excluded from the scope of TALIS. Thus, coverage of TALIS extends to all teachers of ISCED Level 2 and to the principals of the schools where they teach.

An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in programs at ISCED Level 2. In the United States, ISCED Level 2 teachers are those who provide any instruction for grades 7, 8, and/or 9. Teachers who teach a mixture of programs at different levels including ISCED Level 2 programs in the target school are included in the TALIS universe, as well as teachers who engage with individual or small groups of students in “pull in” or “push out” programs. There is no minimum cut-off for how much ISCED Level 2 teaching—that is, either the number of classes or students-- these teachers need to be engaged in to be included.

The international target population of TALIS restricts the survey to those teachers who teach regular classes in ordinary schools and to the principals of those schools. Teachers teaching to adults and teachers who teach in special needs-only schools are not part of the international target population and are deemed “out of scope.” Special education teachers working in regular public and private schools are eligible to participate.

When schools are comprised exclusively of special education teachers, the school itself is said to be “out of scope.” Teacher aides, pedagogical support staff (e.g. guidance counselors, librarians), and health and social support staff (e.g. doctors, nurses, psychiatrists, psychologists, occupational therapists, and social workers) were not considered as teachers and thus not part of the TALIS international target population.

For national reasons, some participating countries will chose to restrict the coverage of their national implementation of TALIS to parts of the country. For example, a province or state experiencing civil unrest or an area struck by a natural disaster could be removed from the international target population to create a national target population. Participating countries were encouraged to keep these exclusions to a minimum.

Ideally, all the members of the target population ought to be eligible for sampling and data collection. This is the option that TALIS chose and, as a consequence, the international survey population (those who can be surveyed) is identical to the international target population (those who should be surveyed).

TALIS recognizes that attempting to survey teachers in very small schools, those in schools with no more than three teachers at ISCED Level 2, and those teaching in schools located in geographically remote areas could be a costly, time-consuming, and statistically inefficient exercise. Therefore, participating countries are allowed to exclude those teachers for TALIS data collection, thus creating a national survey population different from the national target population. The National Project Manager (NPM) is required to document the reasons for exclusion, the size, the location, the clientele, and so on for each excluded school. Moreover, as discussed later in this section, during data collection in the selected schools, some teachers may be excused from data collection.

Ultimately, samples of schools and teachers will be selected from the national survey population.

Within a selected in-scope school, the following teachers are to be excluded: teachers who also act as principals (no teacher data collected, but principal data collected); substitute, emergency, or occasional teachers; teachers on long-term leave; and teachers teaching exclusively to adults.

Sampling Frame of Schools

Participating countries provided Statistics Canada (the international sampling contractor) with a current and complete list of schools providing education at ISCED Level 2. This list constitutes the school sampling frame for TALIS and is expected to correspond to the survey population as defined and described on the Sampling Forms. The sampling frame for the United States was derived from the Common Core of Data (CCD) and Private School Survey (PSS) datasets. For the main study, the United States first stage sampling frame (for schools) was based on the 2010-2011 CCD and 2009-2010 PSS datasets, which are the most recent versions available of these datasets.

The sampling frame contained certain key variables: the type of funding (private or public) and grade structure, which are the explicit strata; a national school identifier; a measure of size (MOS) corresponding to the number of ISCED Level 2 teachers; values for the variables used for implicit stratification; and whenever possible, the type of education stream (academic or vocational) also appeared in the database for the frame. Additional sampling frames are required for the second stage sampling of teachers, namely, the list of eligible ISCED Level 2 teachers in each selected school.

The sampling frame for the main study used two explicit strata: public/private and grade structure. The grade structure is defined with the following categories:

1. Middle-Junior: Middle School (grades 6 to 8) or Junior High (grades 7 to 9, or grades 7 and 8)

2. High School (grades 9 to 12)

3. Other (any other grade structure that includes at least one ISCED Level 2 grade

The original sampling specifications for selecting the schools for the main study specified the following implicit stratification variables: region (Northeast, Midwest, South, and West), urbanicity (city, suburb, town, rural), percent minority students and number of ISCED Level 2 teachers (measure of size). Within each explicit stratum the schools were sorted by a hierarchical combination of the explicit stratum variables, in order to improve the representativeness of the sample across these variables. In the final sample implementation the urbanicity variable was inadvertently dropped from the implicit stratification sort. After a review of the distribution of the sample schools, it was decided that the other implicit stratification variables were sufficiently effective, so it would not be necessary to reselect the sample to include urbanicity in the implicit stratification.

Table 1 presents the distribution of the eligible schools in the combined main study sampling frame by explicit strata (public/private and grade structure).

Table 1. Distribution of Eligible Schools in TALIS Main Study Sampling Frame by Public/Private and Grade Structure Strata

Grade Structure	Public	Private	Total
1 - Middle-Junior	9,788	80	9,868
2 - High School	11,248	1,126	12,374
3 - Other	15,086	6,908	21,994
Total	36,122	8,114	44,236

Given the small number of private schools with a Middle-Junior grade structure, this stratum was collapsed (combined) with the private schools with the High School grade structure. During recent consultations with the OECD, and Statistics Canada, which is responsible for TALIS sampling at the international level, it was determined that the sample size could be reduced to approximately 200 schools and a minimum of 3,400 teachers in grades 7 to 9. The sample schools were allocated to the different explicit strata proportionally to the total number of ISCED Level 2 teachers. Given the small proportion of the schools in the combined private Middle-Junior and High School stratum, the proportional allocation for this combined stratum was increased from 3 to 4 schools, resulting in a final sample of 201 schools.

At the first sampling stage the schools were selected within each stratum systematically with PPS, where the measure of size (MOS) was based on the estimated number of ISCED Level 2 teachers. Since the number of ISCED Level 2 teachers was not available in the CCD and PSS databases, it was necessary to estimate the approximate number of teachers based on the proportion of the total students in each school who attended grades 7 to 9, multiplied by the total number of teachers. In the case of the schools with more than 3 and less than 20 teachers, the MOS was changed to the average number of teachers for these schools within the explicit stratum. This was the equivalent of selecting the schools in this group with equal probability within each stratum. This was done in order to stabilize the weights, since all ISCED 2 Level teachers in these schools will be selected at the second sampling stage with certainty.

Table 2 shows the distribution of the 201 main study sample schools by explicit stratum (public/private category and grade structure).

Table 2. Distribution of Eligible Sample Schools Selected for TALIS Main Study by Public/Private and Grade Structure Strata

Grade Structure	Public	Private	Total
1 - Middle-Junior	71	0	71
2 - High School	49	4	53
3 - Other	63	14	77
Total	183	18	201

It is important to maintain the effective number of sample schools, so any non-responding school will be replaced by a pre-selected similar school. This will be implemented by selecting two potential replacements, the school preceding the sample school in the sampling frame sorted by implicit stratum as well as the one following the sample school. These are designated as the first and second replacement schools that will be available in case the original sample school does not respond. This sampling strategy—of having two replacement schools for each original school—is used in other international education studies such as PIRLS, PISA, and TIMSS.

Teacher Samples

To allow for reliable estimation and modeling, while taking into account the expected levels of non-response, the sample size for the TALIS main study is set at 22 ISCED Level 2 teachers within each participating school, or all of the eligible teachers when the school has less than 22. In schools with more than 22 eligible teachers, a random sample of 22 eligible teachers will be drawn. By varying the sampling rate of teachers in this way, it is anticipated that the achieved sample of teachers will be as close to 3,400 as possible without over-burdening schools. A sample of 201 schools was drawn from the population of in-scope schools, which is consistent with the TALIS international sampling specifications. Based on calculations which estimate the total number of TALIS-eligible faculty at these schools, it is anticipated that this sampling strategy will result in a sample of at least 3,500 teachers (before refusals), which based on the experience from the previous TALIS will provide a sufficient level of precision for the analysis (after refusals).

To sample ISCED Level 2 teachers within each sampled school, the international procedures call for the designation of a school coordinator in each school. In addition to assistance with arranging for the study within each school, the school coordinator of each school is asked to prepare a list of all eligible teachers in the school (using a standardized Teacher Sampling Form (TSF)). In the United States, eligible teachers are those teachers who instruct at least one student at the 7^th, 8^th, or 9^th-grade levels, regardless of subject matter taught. As long as a teacher instructs at least one student within this grade range, s/he is considered an ‘eligible teacher.’ Any teacher who does not teach any students at these grade levels is out of scope and will not be sampled for TALIS.

The TSF includes: identification number (number generated by sampling software); name of teacher; birth year; sex; main teaching domain (main subject taught); exclusion status (i.e., excluded or not); and email address. Each completed TSF is submitted to NCES (or its national contractor). This information is entered into the sampling software by NCES or its national contractor (WinW3S, proprietary sampling software developed by the international contractor and used in other international studies such as TIMSS and PIRLS). Based on the list of teachers as well as the total number of eligible ISCED Level 2 teachers in each school, teachers will be selected to participate. The information collected from the TSF remains with NCES and/or its national contractor. The names and email addresses of teachers are not associated with the TALIS database in any way; these are collected to allow for follow-up, should it be necessary. Once it is determined that no further follow-up is needed, all TSFs will be destroyed.

Nonresponse Bias Analysis, Weighting, Sampling Errors

In order to meet international standards for participation, for the main study the United States must achieve minimum response rates established by OECD and also meet OMB and NCES standards. While the OECD has set a minimum response rate of 75 percent of sampled schools (after replacement), NCES’s response rate standards apply to each data collection unit (in this case, schools and teachers) and are calculated WITHOUT replacements. NCES Statistical Standard 2-2-2 indicates that for a cross-sectional survey (without an assessment), the target response rate is at least 85 percent for each sampling unit (see http://nces.ed.gov/statprog/2002/std2_2.asp).

At the teacher level, the OECD has set a minimum response rate of 75 percent of selected teachers (in both original and replacement schools). Participating schools that fail to yield at least a 50 percent response rate among the sampled teachers will be considered as non-participating, even though the number of participating teachers may be enough to contribute to some of the analyses. NCES standards require a minimum response rate of 85 percent of sampled teachers.

It is inevitable that nonresponse will occur at both levels: school and teacher. NCES and its national contractor will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, at minimum complying with OMB and NCES Statistical Standards. According to the standards, a nonresponse bias analysis (NRBA) must be conducted if either the school or teacher response rate falls below this threshold (main study only). After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification. Although the NRBA will include a comparison of frame variables for respondents and nonrespondents, the specific nature of the analysis and how extensive that analysis is will be a function of the amount of nonresponse combined with the results from the initial frame analysis.

As part of the NRBA, two forms of analysis are usually undertaken: a test of the independence of each participating individual’s characteristics and participation status, and a logistic regression in which the conditional independence of selected individual characteristics as predictors of participation is examined. This analysis is required by NCES to support a decision to include U.S. results in the international report in the event the response rate is below 85 percent. The report may also be required by OECD should U.S. response rates fall below the minimum of 75 percent of schools or teachers. An example of a recent non-response bias analysis can be found in chapter 3, section 3.4 of the TIMSS 2007 Technical Report at http://nces.ed.gov/pubs2009/2009012.pdf.

In the event that recruitment efforts indicate that response rates will likely be below the minimum threshold set by NCES, the contractor is required to submit an outline for the non-response bias analysis report 1 month prior to data collection, or as soon as it becomes clear that a non-response bias analysis will be needed. A preliminary report will be due within 2 months of the completion of data collection so that results of the non-response bias analysis can be used in pre-release decisions about U.S. sample coverage. The contractor shall produce weights to be used in this analysis because weights produced by the International Consortium will not be available until a later date. An updated version of the non-response bias analysis report must be produced after weights are provided by the International Consortium. This report must be reviewed and approved by the NCES Chief Statistician. A reduced form of the final report (a summary of findings), with final data and weights provided by the International Consortium, also must be included in the national report (to be included in the report’s technical appendix). The complete, approved non-response bias analysis report will be included in the data file user’s guide.

B.3 Maximizing Response Rates

Our approach to maximizing school and teacher response rates in the main study includes the following:

• Selecting and notifying schools beginning in early September 2012;

• Approaching schools directly, and notifying states and school districts;

• Assigning personal recruiters for specific schools;

• Incentives for schools, school coordinators, and teachers (see Section A9);

• Contact with schools and school coordinators at set intervals throughout the months preceding the survey; and

• Collection of data in Jan-April 2013 (Main Study) to avoid conflicts with state testing which normally occurs in late spring/end of school year.

B.4 Purpose of Main Study Data Uses

The Teaching and Learning International Survey (TALIS) offers the opportunity for teachers and principals to provide input into education analysis and policy development. TALIS is being conducted by the Organization for Economic Cooperation and Development (OECD). Cross-country analysis of these data will allow countries to identify other countries facing similar challenges and to learn from other policy approaches. School principals and teachers will provide information about issues such as the professional development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and recognition they receive about their work; and various other school leadership, management, and workplace issues. In the TALIS study, it is the intention to draw a picture of the different workforce issues and educational practices in all the participating countries. Countries and individuals may differ in this regard.

The data from the TALIS main study will be reported via an international report that will be released in June 2014. NCES and its national contractor intend to produce a U.S. national report based on the TALIS main study data received from the international sponsors. The U.S. national report will be released at the same time as the international report from the OECD. This report will be tailored to a general audience and will report results of the United States relative to other countries.

B.5 Individuals Consulted on Study Design

Many people at OECD, Statistics Canada, and other organizations around the world have been involved in the design of TALIS. Overall direction for TALIS in the U.S. is provided by Patrick Gonzales at the National Center for Education Statistics, the TALIS Board of Participating Countries representative, and Gregory Strizek, the TALIS National Project Manager and the project co-director at the national contractor Strategic Analytics, Inc.

Patrick Gonzales

National Center for Education Statistics

Institute of Education Sciences

U.S. Department of Education

415-920-9229

[email protected]

Gregory A. Strizek

National Project Manager

Strategic Analytics, Inc.

703-346-4387

[email protected]

OECD Teaching and Learning International Survey
(TALIS) 2013 Main Study

Supporting Statement Part C

National Center for Education Statistics

Institute Of Education Sciences

U.S. Department Of Education

Washington, D.C.

September 7, 2012

INSTRUMENT DEVELOPMENT SUPPORTING DOCUMENTATION

The Teaching and Learning International Survey (TALIS) is an international survey of teachers and principals focusing on the working conditions of teachers and the teaching and learning practices in schools. TALIS has been administered one time previously, in 2008. To prepare for the second TALIS data collection in 2013, the international sponsoring organization, the Organization for Economic Cooperation and Development (OECD), created an Item Development Expert Group (IDEG) to guide the refinement of the 2008 survey instruments as well as the development of new (to TALIS) measures. TALIS is a collaborative effort by the participating countries, guided by their governments on the basis of shared policy driven interests. Representatives of each country form the TALIS Board of Participating Countries (BPC), which determines the policy orientations of TALIS as well as the analysis and results produced from it. Through the National Center for Education Statistics (NCES) of the Institute of Education Sciences, the United States has engaged in the development of the 2013 main study instruments. The TALIS 2013 survey instruments—a school questionnaire for the principal or head administrator and a teacher questionnaire—have undergone revision and refinement in preparation for the main study data collection. Specifically, the United States sought major changes to the field trial versions of the questionnaires in three main areas as well as a complete rewrite of the Framework document which guides the development of the TALIS instruments and study. The three areas of change to the questionnaires sought by the United States included: (1) improve the measure of teacher working time to include time spent on all activities related to the job as well as only those hours spent in actual teaching; (2) improve the measure of teacher educational attainment to make it less complicated; and (3) preserve the measure of social desirability. The United States was successful in goals (1) and (2), but was unable to convince other country representatives to include the measure of social desirability as a requirement of participation.

The social desirability scale, based on a modified form of the Marlowe-Crowne social desirability scale (Crowne and Marlowe 1960¹), was fielded in an effort to better understand and measure possible impression management, both positive and negative, in the responses provided by teachers. While extreme response styles (ERS) and acquiescent response styles (ARS) can be investigated using the main items in the instruments, impression management cannot be investigated without fielding a scale specific to this purpose. Although the modified scale showed fair internal consistency coefficients across all field test countries (Cronbach’s alpha of .65 for positive impression management; .57 for negative impression management; .65 when both taken together), and demonstrated both configural and metric invariance, the scale did not demonstrate scalar invariance, and was dropped as a required component of the international instruments for this reason. However, based on the unpublished field trial data, the scale showed that, as suspected, teachers (at the country level) showed substantial response biases that suggested they were responding as they were expected to rather than how they may really feel. Country-level correlations with a host of proposed TALIS scales (e.g., job satisfaction, school climate) indicated the greatest observed effect on teachers’ constructivist beliefs, teacher and principal job satisfaction, teacher and principal perceptions of school climate, principals’ satisfaction with the profession, and principals’ satisfaction with the current job environment. Based on the field trial data, the modified social desirability scale showed meaningful patterns of cross-cultural differences in response styles and their potential effects on some of the proposed TALIS scales.

Based on a request from the United States, the social desirability scale (#49 in the teacher questionnaire) was offered to countries as a ‘national option,’ allowing for its use in the main study to further the development and validation of this measure. The United States proposes to include the scale in its version of the instruments and has requested that other countries include the social desirability scale for this purpose. Assuming other countries include this option in their national instruments, a cross-country dataset can be created that allows for the further exploration of social desirability in response styles, albeit as a separate activity from TALIS.

After the field trial, both the principal and teacher questionnaires were shortened, with the teacher questionnaire including the most item deletions. In particular, the section on teaching practices in the teacher questionnaire was greatly reduced due to poor performance of the proposed scales in the field trial. At the request of some countries, a number of the individual teaching practice items have been maintained because they remain of policy interest even without the larger scales.

With regard to the Framework document, the United States sought a complete rewrite that incorporates a more balanced discussion of the research literature, including recognition that much of the cited research is conducted solely within a single national context, rather than cross-country, and is often limited to small samples that cannot be easily extrapolated to larger populations. The United States also objected to the repeated invocation of the terms “effective teachers” and “effective teaching,” based on this emerging theme in the research literature. Given that TALIS is based on respondents’ self-reports to a series of questions and does not include any measure of actual student learning, the United States urged the OECD and other countries to be cautious in linking TALIS to the “effective teachers” and “effective teaching” literature and debates. After much discussion and prodding, the position of the United States was recognized as being valid and reasonable. The TALIS Framework is currently undergoing a major revision to address these and other issues. It is hoped that the Framework will be made available to the public in late Fall/early Winter 2012.

In addition to the three primary goals stated above, the United States also sought some wording changes which were considered important to improve the interpretation of items. The wording changes were based on NCES’ experience with SASS, as well as the research of the experts consulted by NCES (i.e., the TALIS steering committee). Some, but not all, of these suggested changes were accepted by the countries and study sponsor. The versions of the instruments submitted in this package (see Appendix A) are the final, adapted versions to be administered in the United States during spring 2013 pending OMB approval.

The principal questionnaire includes sections on principal’s personal background information, school background information, school climate, school leadership, teacher appraisal and feedback, principal continuous professional development, and teacher induction and mentoring. The teacher questionnaire includes sections on teacher’s background information, teacher continuous professional development, teacher appraisal and feedback, mentoring and induction, teaching practices, beliefs and attitudes, school climate, and job satisfaction.

Because the themes explored in TALIS cut across the school and teacher questionnaire, the themes are discussed jointly. The rationale for each of these themes is briefly outlined below.

Theme: personal background information (principal instrument Q1-Q6; teacher instrument Q1-Q19)

This theme focuses primarily on key socio-demographic variables that are fairly standard for data collections of this type. Questions related to a respondent’s gender, age, educational attainment, employment status, and years of experience are considered key factors that may relate to differences found in the other themes explored in TALIS. The teacher questionnaire also includes items related to the respondent’s teacher education program, while the principal questionnaire includes items related to principal continuing professional development.

Theme: school background information (principal instrument Q9-Q15)

This theme includes basic school socio-demographic factors that may relate to differences explored in other themes in TALIS. Questions in this theme relate to the size of the school (student and staff head count), funding sources, school locale, school choice competition, and basic information on student characteristics such as the percentage of English Language Learners, special needs students, and students from disadvantaged homes.

Theme: teacher appraisal and feedback (principal instrument Q27-Q29; teacher instrument Q30-Q33)

This theme focuses on the presence of and processes related to teacher appraisal and feedback, as performed by the school principal and others. As defined in TALIS, teacher appraisal and feedback occurs when a teachers’ work is reviewed by the school principal, an external inspector, or a teacher’s colleagues. Such evaluations provide an opportunity for teachers to receive feedback on their work and can serve as a way to identify what is and is not working in the classroom.² As learned through TALIS 2008, the specific form of appraisal can vary widely across contexts and can be formal or informal, subjective or objective, and summative or formative. Research suggests that summative evaluation systems may be useful for quality assurance and accountability purposes as well as teacher recognition while formative evaluation systems can help teachers further develop their pedagogical practices.³ Research also suggests that it is important that appraisal and feedback systems be viewed as an integrated part of the school culture rather than an “add-on.”⁴ Depending on the form of teacher appraisal and feedback, evaluation can have a significant impact on elements of teachers’ professional lives, including continuing professional development, job satisfaction, and compensation. Based on TALIS 2008 data, teacher appraisal and feedback is often only weakly linked to teacher professional development opportunities and plans but teachers reported that appraisal and feedback increased their job satisfaction.⁵ Because teachers’ level of job satisfaction has been found to have an impact on the classroom environment and student outcomes in some contexts,⁶ the kinds of evaluation and feedback systems in place in schools may have an indirect affect on student learning.

Theme: school climate (principal instrument Q30-Q32; teacher instrument Q46-47)

School climate in TALIS refers broadly to the quality and character of school life, and has been shown to be related to student well-being and academic achievement.⁷ The theme of school climate is intended to provide indicators about teacher-parent and teacher-student relations, factors shown in some research to be related to student achievement.⁸ Although the definition of school climate varies across studies and contexts, research has identified several factors that can contribute to a positive school climate. Among these are teaching and learning practices, disciplinary norms, decision-making processes, organizational structures, safety, a sense of community, and interpersonal relationships.⁹ Of these issues, TALIS focuses primarily on disciplinary issues, organizational structures, and community/interpersonal relationships.

Results of some studies suggest a relationship between student achievement and school organization. For example, one study found that measures of professional culture in schools—including three dimensions: satisfaction with leadership, professional cooperation, and consensus—were related to levels of bullying found in the schools: schools with the highest levels of bullying had significantly worse scores on these professional culture measures than schools with the lowest levels of bullying.¹⁰ Another study found that a safe and stimulating learning environment had a positive effect on student involvement, attitudes, and educational attainment.¹¹ Further, research suggests that the impact of a positive school climate may be important for “at risk” and immigrant students.¹² School climate has been found to also affect teachers, including teacher retention.¹³

Theme: school leadership (principal instrument Q16-Q26)

This theme focuses on identifying factors associated with different styles of school leadership, whether it be a more authoritative, top-down approach or a shared, distributed approach. As found in TALIS 2008, different styles of school leadership appear to play a role in teachers’ working lives and can help develop the ability of teachers to support student achievement.¹⁴ And, as noted in a report on PISA 2009 results, “the quality of an education system cannot exceed the quality of its teachers and principals.”¹⁵ The IDEG and BPC thus considered it important to more closely examine the characteristics of school leadership, how leadership may impact both teachers and students, and how leadership is cultivated in the workplace. In TALIS, school leadership has been expanded beyond the sole responsibility of the school principal to include the possibility of distributed or team leadership within schools. Two core elements of a principal’s role or style of leadership included in TALIS are instructional leadership and administrative management. Instructional leadership is defined as the actions that a principal takes to promote growth in student learning.¹⁶ Research suggests that principals who take a strong role in instructional leadership emphasize high-quality instruction and develop policies that support student achievement, such as supporting the development of learning communities, giving instructional feedback to teachers, modeling instructional practices, and supporting the use of assessment data in the classroom.¹⁷ In addition to taking a leadership role in the area of instruction, research suggests that successful school leaders develop the organization by strengthening the school’s collaborative culture and creating a more flexible organization structure by demonstrating participatory or distributed leadership.¹⁸ Distributed leadership in school is thought to result in a work culture that promotes collegiality and collaborative problem solving, both of which have been shown to be related to student achievement by affecting key teacher variables such as teacher motivation and teacher capacity.¹⁹ Of course, successful distributed leadership does not spontaneously occur; rather, it is thought to result from collective discussion, development, and dialogue among those working within the school.²⁰

Theme: teacher and principal continuous professional development and teacher mentoring and induction (teacher instrument Q20-29; principal instrument Q7-Q8 and Q33-38)

This theme focuses primarily on teacher education, from initial education through induction to in-service professional development and mentoring. Research suggests that teacher quality has implications for student success.²¹ In some countries, schools attempt to support professional development by allowing professional development activities to be organized during work hours and permitting teachers to obtain a leave of absence or a research grant to undertake studies. Results from TALIS 2008 indicated, however, that many teachers report unmet professional development needs, particularly in the areas of teaching special needs students, teaching with technology, and student discipline.²² Although most teachers indicated that they participated in some form of professional development in 2008, the majority of teachers reported they participated in professional development activities for less than one day per month. Given the gap between teachers’ reported professional development needs and the training they are receiving, TALIS 2013 seeks to examine in more detail the practice, needs, and limitations to pursuing professional development.

It is thought that high-quality professional development, particularly in the form of mentoring or induction, may increase the likelihood of retaining teachers in the profession, a concern shared by a number of countries.²³ A number of studies suggest that mentoring is effective for increased retention and stability in schools and can result in changes in teaching practices.²⁴

Teacher continuing professional development has also been shown to impact teachers’ classroom practices.²⁵ However, its impact appears varied, as the research literature suggests it may have the strongest impact on changes in teacher learning, followed by changes in teacher behavior, and less of an impact on student learning.²⁶ Research also suggests that professional development that occurs in a continuous fashion (more than one-day workshops) and at regular intervals may be most effective for supporting changes in teachers’ practices.²⁷ TALIS seeks to examine in more detail how schools implement teacher continuing professional development, particularly in an era of increased accountability and student diversity.²⁸

Four types of professional development have been found to have the most impact on teacher knowledge and behavior: observation of actual classroom methods; microteaching; video/audio feedback; and practice.²⁹ Research further suggests that three core features of teacher professional development can have positive effects on teachers’ self-reported increases in knowledge and skills and can lead to change in classroom practices: collaboration and active learning; continuity across time and activities; and differentiation.³⁰

TALIS also seeks limited information from principals on the types and frequencies of professional development available to them (PQ7-8). This is an area that has received far less attention than teacher continuing professional development, an area to which TALIS hopes to contribute.

Theme: teaching practices, beliefs, and attitudes (teacher instrument Q34-Q45)

Though there are challenges in studying teachers’ beliefs due to their subjective nature, a body of research has developed which suggests that teachers’ beliefs are related to classroom practices and, ultimately, what students learn.³¹ As one component in the process of teaching and learning, teachers’ instructional or pedagogical beliefs contribute to a deeper, multifaceted understanding about the context of a successful learning environment. The link between beliefs and practices is fairly well established.³² Research also shows that of the many factors associated with teaching and learning, teachers’ instructional beliefs are relatively malleable,³³ although triggering changes in beliefs remains a challenge.³⁴

Evidence from TALIS 2008 suggests that significant variation in beliefs about instruction exists at the country- and teacher-level.³⁵ That is, teachers within a given school tend to hold different beliefs about instruction as do teachers across countries. It might be unsurprising that meaningful differences in instructional beliefs were found across countries, since language, culture, geography, and values all contribute to the school climate and might influence the belief systems of teachers within a country. Less expected, however, is the relative heterogeneity of teachers’ instructional beliefs within schools. One possible explanation for this finding is that teachers’ beliefs are formed early on and remain stable over time. TALIS 2013 will continue to examine variation in teachers’ instructional beliefs at the country, school, and teacher levels.

In addition to understanding more about teachers’ instructional beliefs, TALIS also aims to measure teachers’ beliefs about student assessment practices since the global movement toward assessment and accountability continues to gain momentum and traction with policy makers.³⁶ Previous research indicated that, especially with respect to accountability assessments, teachers felt confused and powerless and that testing negatively impacted teacher practices.³⁷ Research has shown that teachers generally believe that assessment for accountability is more harmful than good and contributes to feelings of anxiety, shame and guilt on the part of the teacher.³⁸ As such, it will be useful to policy makers and researchers to have an international perspective on the way teachers think about assessment and how this relates to learning environments.

Pedagogical practices are at the core of teachers’ and students’ classroom experiences, yet TALIS 2008 revealed that in some countries, including Mexico, Italy, and Turkey, school leaders reported that a lack of pedagogical preparation hindered the provision of instruction.³⁹ Research suggests that it is important to offer teachers opportunities not only to develop their classroom practices, but their pedagogical practices as well. For example, research suggests that when teachers have opportunities to expand and develop their own teaching repertoires, they are more likely to provide an increased range of learning opportunities for students.⁴⁰ Similarly, in a study of special education teachers recognized for excellence in teaching, teachers who were effectively able to promote literacy development appeared to have experience with a variety of instructional approaches and have extensive background knowledge about instructional practices.⁴¹

It is important to acknowledge that although successful pedagogical practices overlap across subjects and student populations, some practices may be specific to particular subjects or student populations. For example, for mathematics teachers, research into the benefits of teacher-directed and student-centered instruction remains largely inconclusive, though there is evidence that well designed and effectively implemented tutorials can have a positive impact on mathematics performance, particularly at the middle and high school levels.⁴² Physical resources also appear to impact student learning in mathematics, but results are somewhat inconclusive.⁴³ For example, a study of 8th grade students found that students in classes that use calculators regularly score higher on the National Assessment of Educational Progress than students in classes where calculators are used rarely if at all.⁴⁴ At the same time, a review of eleven recent studies found that calculators have limited to no impact on calculation skills, problem solving, or conceptual development over periods of up to one year, though the impact of long-term use was not examined.⁴⁵

Teacher practices are more than just pedagogical in nature; they also include professional practices. Studies of teachers focus not only on what happens in the classroom, but also on the professional activities in which teachers are involved, including the ways in which teachers interact with one another.⁴⁶ Co-operation and collaboration among teachers has repeatedly been found to be a particularly important element of professional practice and can include the exchange of instructional materials, developing curricula, meeting to discuss student progress, and collective learning activities.⁴⁷ Some researchers have argued the importance of pedagogical practices for student achievement warrants a shift in initial teacher education from a focus on what teachers know and believe to focus on what teachers do.⁴⁸

SUMMARY

The instruments developed for TALIS 2013 include items administered in 2008 as well as new or newly-revised items for 2013. In cooperation with other participating countries, and through interaction and discussion with experts in the areas of teachers, teaching, and workforce issues, the TALIS 2013 instruments include improved measures that will increase the utility of the dataset and provide better indicators on which policymakers and educators can make critical decisions. The effort to improve the measures in TALIS will not end with the 2013 administration; rather, as TALIS moves forward and matures as a regularly instituted study, measures will be refined or added as more is learned from high quality research conducted in the United States and elsewhere. In this way, it is expected that TALIS will become an important policy and research endeavor that will greatly contribute to our understanding of schooling and learning.

Non-response bias analysis

In order to meet international standards for participation, for the main study the United States must achieve minimum response rates established by OECD and also meet NCES standards. While the OECD has set a minimum response rate of 75 percent of sampled schools (after replacement), NCES has its own standards regarding response rates. NCES response rates for each data collection unit (in this case, schools and teachers) are calculated WITHOUT replacements. NCES Statistical Standard 2-2-2 indicates that for a cross-sectional survey (without an assessment), the target response rate is at least 85 percent for each sampling unit (see http://nces.ed.gov/statprog/2002/std2_2.asp).

At the teacher level, the OECD has set a minimum response rate of 75 percent of selected teachers (in both original and replacement schools). Participating schools that fail to yield at least a 50 percent response rate among the sampled teachers will be considered as non-participating, even though the number of participating teachers may be enough to contribute to some of the analyses. NCES standards require a minimum response rate of 85 percent of sampled teachers.

In the event that the NCES standard of 85 percent response rate per sampling unit is not met, the contractor will be required to conduct a non-response bias analysis to explore the extent of potential bias introduced into U.S. study data through non-response on the part of sampled schools and teachers. The analyses that the contractor will conduct compares selected characteristics likely to reflect bias in participation between participating and non-participating schools and teachers. Two forms of analysis are usually undertaken: a test of the independence of each participating individual’s characteristic and participation status, and a logistic regression in which the conditional independence of selected individual characteristics as predictors of participation is examined. This analysis is required by NCES to support a decision to include U.S. results in the international report. The report may also be required by OECD should U.S. response rates fall below the minimum of 75 percent of schools or teachers. An example of a recent non-response bias analysis can be found in chapter 3, section 3.4 of the TIMSS 2007 Technical Report at http://nces.ed.gov/pubs2009/2009012.pdf.

In the event that recruitment efforts indicate that response rates will likely be below the minimum threshold set by NCES, the contractor will submit an outline for the non-response bias analysis report 1 month prior to data collection, or as soon as it becomes clear that a non-response bias analysis will be needed. A preliminary report will be due within 2 months of the completion of data collection so that results of the non-response bias analysis can be used in pre-release decisions about U.S. sample coverage. The contractor shall produce weights to be used in this analysis because weights produced by the International Consortium will not be available until a later date. A reduced form of the preliminary report shall be included in the technical appendix of the national report. An updated version of the non-response bias analysis report must be produced after weights are provided by the International Consortium. This report must be reviewed and approved by the NCES Chief Statistician. A reduced form of the final report (a summary of findings), with final data and weights provided by the International Consortium, also must be included in the national report (to be included in the report’s technical appendix).

The complete, approved non-response bias analysis report will be included in the datafile user’s guide.

1 Crowne, D.P. and Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349-354.

2 Behn, R.D. (2003). Why measure performance? Different purposes require different measures. Public Administration Review, 63: 586-606.

3 Santiago, P. and Benavides, F. (2009). Teacher evaluation: A conceptual framework and examples of country practices. Paris: OECD. Retrieved from http://www.oecd.org/dataoecd/16/24/44568106.pdf.

Shinkfield, A.J. and Stufflebeam, D.L. (1996). Teacher evaluation: Guide to effective practice. Norwell, MA: Kluwer Academic Publishers.

4 Marhsall, K. (2005). It’s time to rethink teacher supervision and evaluation. Phi Delta Kappan, 86(10), 727-735.

5 Organization for Economic Cooperation and Development. (2009). Creating Effective Teaching and Learning Environments: First Results from TALIS. Paris: Author.

6 Michaelowa, K. (2002). Teacher job satisfaction, student achievement, and the cost of primary education in Francophone Sub-Saharan Africa. Hamburg: Hamburg Institute of International Economics.

7 Hattie, J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement. New York: Routledge.

8 Cornelius-White, J. (2007). Learner-centered teacher-student relationships are effective: A meta analysis. Review of Educational Research, 77(1), 113-143.

9 Anderson, C.S. (1982). A search for school climate: A review of the research. Review of Educational Research, 52, 368.

Cohen, J., McCabe, E.M., Michelli, N.M., and Pickeral, T. (2009). School climate: research, policy, practice, and teacher education. Teachers College Record 111(1), 180-213.

Brophy, J. (1988). Research linking teacher behavior to student achievement: Potential implication for instruction of Chapter I students. Educational Psychologist, 23(3), 235-286.

Engel, L., Rutkowski, D., and Rutkowski, L. (2009). The harsher side of globalization: Violent conflict and academic achievement. Globalisation, Societies and Education, 7(4), 433-456.

10 Roland, E., and Galloway, D. (2004). Professional cultures in schools with high and low rates of bullying. School Effectiveness and School Improvement, 15(3-4), 241-260.

11 Van de Grift, W. (2007). Quality of teaching in four European countries: A review of the literature and application of an assessment instrument. Educational Research, 49(2), 127-152.

12 Downer, J.T., Rimm-Kaufman, S.E., and Pianta, R.C. (2007). How do classroom conditions and children’s risk for school problems contribute to children’s behavioral engagement in learning? School Psychology Review, 36(3), 413-432.

Den Brok, P., van Tartwijk, J., Wubbels, T., and Veldman, I. (2010). The differential effect of the teacher-student interpersonal relationship on student outcomes for students with different ethnic backgrounds. British Journal of Educational Psychology, 80(2), 199-221.

13 Battistich, V., Solomon, D., Watson, M., and Shaps, E. (1997). Caring school communities. Educational Psychologist, 32(3), 137-151.

Miller, M.D., Brownell, M.T., and Smith, S.W. (1999). Factors that predict teachers staying in, leaving, or transferring from the special education classroom. Exceptional Children, 65(2), 201-218.

14 Organization for Economic Cooperation and Development. (2009). Creating Effective Teaching and Learning Environments: First Results from TALIS. Paris: Author.

15 Organization for Economic Cooperation and Development. (2010). PISA 2009 Results: What Makes a School Successful? Resources, Policies and Practices (Volume IV). Paris: Author.

16 Sergiovanni, T.J., Kelleher, P., McCarthy, M., and Fowler, F.C. (2009). Educational Governance and Administration (6^th edition). Boston: Allyn and Bacon.

17 Blase, J., and Blase, Jo. (200). Effective instructional leadership: Teachers’ perspectives on how principals promote teaching and learning in schools. Journal of Educational Administration, 38(2), 130-41.

National Assocation of Elementary School Principals. (2001). Leading Learning Communities: Standards for What Principals Should Know and Be Able to Do. Alexandria, VA: Author.

18 Barker, B. (2007). The leadership paradox: Can school leaders transform student outcomes? School Effectiveness and School Improvement, 18(1), 21-43.

Daly, A. (2009). Rigid response in an age of accountability: The potential of leadership and trust. Educational Administration Quarterly, 45, 168.

19 Leithwood, K., and Mascall, B. (2008). Collective leadership effects on student achievement. Educational Administration Quarterly, 44(4), 529-561.

Hallinger, P., and Heck, R.H. (2010). Leadership for learning: Does collaborative leadership make a difference in school improvement? Educational Management Administration and Leadership, 38(6), 654-678.

20 Harris, A. (2002). School Improvement: What’s in it for Schools? London: RoutledgeFalmer.

21 Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy evidence. Seattle, WA: University of Washington, Center for the Study of Teaching and Policy.

Heyneman, S.P. and Loxley, W. (1983). The distribution of primary school quality within high and low-income countries. Comparative Education Review, 27, 108-118.

22 Organization for Economic Cooperation and Development. (2009). Creating Effective Teaching and Learning Environments: First Results from TALIS. Paris: Author.

23 Smith, T.M., and Ingersoll, R.M. (2004). What are the effects of induction and mentoringon beginning teacher turnover? American Educational Research Journal, 41(3), 681-714.

Dove, M.K. (2004). Teacher attrition: A critical American and international education issue. Delta Kappa Gamma Bulletin, 71(1), 8-30.

Macdonald, D. (1999). Teacher attrition: A review of the literature. Teaching and Teacher Education, 15(8), 835-848.

24 Hobson, A.J., Ashby, P., Malderez, A., and Tomlilnson, P.D. (2009). Mentoring beginning teachers: What we know and what we don’t. Teaching and Teacher Education, 25, 207-216.

Strong, M., Villar, A., and Fletcher, S. (2008). An investigation of the effects of variations in mentor-based induction on the performance of students in California. Teachers College Record, 110(10), 2271-2289.

25 Clotfelter, C.T., Ladd, H.F., and Vigdor, J.L. (2007). How and why do teacher credentials matter for student achievement? Sanford Institute, Dale University, Calder Working Paper 2.

26 Hattie, J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement. New York: Routledge.

27 Ball, D.L. (1996). Teacher learning and the mathematics reforms: What do we think we know and what do we need to learn? Phi Delta Kappan, 77, 500-508.

Garet, M.S., Porter, A.C., Desimone, L., Birman,, B.F., and Yoon, K.S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915-945.

Kennedy, A. (2011). Collaborative continuing professional development (CPD) for teachers in Scotland: Aspirations, opportunities and barriers. European Journal of Teacher Education, 34(1), 25-41.

Wei, R.C., Darling-Hammond, L., Andree, A., Richardson, N., and Orphanos, S. (2009). Professional learning in the learning profession: A status report on teacher development in the U.S. and abroad. Dallas, TX: National Staff Development Council.

Supovitz, J., Mayer, D.P., and Kahle, J.B. (2000). Promoting inquiry-based instructional practice: The longitudinal impact of professional development in the context of system reform. Educational Policy, 14, 331-356.

28 Saha, L.J. and Dworkin, G.A. (2009). Introduction: New perspectives on teachers and teaching. In L.J. Saha and A.G. Dworkin (Eds.), International Handbook of Research on Teachers and Teaching (pp. 3-11). New York: Springer.

Organization for Economic Cooperation and Development. (2010). PISA 2009 Results: What Makes a School Successful? Resources, Policies, and Practices (Vol IV). Paris: Author.

29 Hattie, J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement. New York: Routledge.

30 Garet, M.S., Porter, A.C., Desimone, L., Birman,, B.F., and Yoon, K.S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915-945.

31 Wilson, M., and Cooney, T. (2003). Mathematics teacher change and development: The role of beliefs. In G.C. Leder, E. Pehkonen, and G. Torner (Eds.), Beliefs: A Hidden Variable in Mathematics Education? (pp. 127-147). Doredrecht, the Netherlands: Kluwer.

Hoy, A.Q., Davis, H., and Pape, S.J. (2006). Teacher knowledge and beliefs. In P.A. Alexander and P.H. Winne (Eds.), Handbook of Educational Psychology (2^nd edition) (pp. 715-737). Mahwah, NJ: Earlbaum.

Staub, E., and Stern, E. (2002). The nature of teachers’ pedagogical content beliefs matters for students’ achievement gains: Quasi-experimental evidence for elementary mathematics. Journal of Educational Psychology, 94, 344-355.

32 Muijs, D., and Reynolds, D. (2001). Teacher beliefs and behavior: What really matters. Journal of Classroom Interaction, 37, 3-15.

Wilson, M. and Cooney, T. (2003). Mathematics teacher change and development: The role of beliefs. In G.C. Leder, E. Pehkonen, and G. Torner (Eds.), Beliefs: A Hidden Variable in Mathematics Education? (pp. 127-147). Doredrecht, the Netherlands: Kluwer.

Schunk, D.H. and Zimmerman, B.J. (2006). Competence and control beliefs: Distinguishing the means and ends. In P.A. Alexander and P.H. Winne (Eds.), Handbook of Educational Psychology (2^nd edition) (pp. 349-367). Mahwah, NJ: Earlbaum.

33 Wilson, M. and Cooney, T. (2003). Mathematics teacher change and development: The role of beliefs. In G.C. Leder, E. Pehkonen, and G. Torner (Eds.), Beliefs: A Hidden Variable in Mathematics Education? (pp. 127-147). Doredrecht, the Netherlands: Kluwer.

34 Forgasz, J.J. and Leder, G.C. (2008). Beliefs about mathematics and mathematics teaching. In P. Sullivan and T. Wood (Eds.), The International Handbook of Mathematics Teacher Education, Volume 1: Knowledge and Beliefs in Mathematics Teaching and Teaching Development (pp. 173-192). Rotterdam: Sense Publishers.

35 Organization for Economic Cooperation and Development. (2009). Creating Effective Teaching and Learning Environments: First Results from TALIS. Paris: Author.

36 Rutkowski, L. and Rutkowski, D. (2010). Getting it ‘better’: The importance of improving background questionnaires in international assessment. Journal of Curriculum Studies, 42(3), 411-430.

Rizvi, F. and Lingard, B. (2009). Globalizing education policy. London: Routledge.

37 Brown, D. (1993, April). The political influence of state-mandated testing reform through the eyes of principals and teachers. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA. Retrieved from http://www.eric.ed.gov/PDFS/ED360737.pdf.

38 Haney, W. (2000). The myth of the Texas miracle in education. Education Policy Analysis Archives, 8(41). Retrieved from http://epaa.asu.edu/epaa/v8n41.

39 Organization for Economic Cooperation and Development. (2009). Creating Effective Teaching and Learning Environments: First Results from TALIS. Paris: Author.

40 Harris, A. (2002). School Improvement: What’s in it for Schools? London: RoutledgeFalmer.

41 Rankin-Erickson, J. and Pressley, M. (2000). A survey of instructional practices of special education teachers nominated as effective teachers of literacy. Learning Disabilities Research and Practice, 15(4), 206-225.

42 National Mathematics Advisory Panel. (2008). Foundations for Success: Final Report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.

43 Neuschmidt, O., Hencke, J., Rutkowski, L., and Rutkowski, D. (2010). Effective schools in the Arab region: A multi-level approach using TIMSS 2003 data. In D. Sharpes (Ed.), Handbook on International Studies in Education (pp. 35-58). Charlotte, NC: Information Age Publishing.

44 Sowder, J.T., Wearne, D., Martin, W.G., and Strutchens, M. (2004). What do 8^th grade students know about mathematics? Changes over a decade. In P. Kloosterman and F.K. Lester, Jr. (Eds.), Results and Interpretations of the 1990 through 2000 Mathematics Assessments of the National Assessment of Educational Progress (pp. 105-191). Reston, VA: National Council of Teachers of Mathematics.

45 National Mathematics Advisory Panel. (2008). Foundations for Success: Final Report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.

46 Darling-Hammond, L., Holtzman, D.J., Galin, S.J., and Heilig, J.V. (2005). Does teacher preparation matter? Evidence about teacher certification, Teach for America, and teacher effectiveness. Education Policy Analysis Archives, 13(42).

47 Ying, D. (2007). Teacher educators’ collaborative inquiry in a context of educational innovation in China—A case study of RICH as a learning community. In T. Townsend and R. Bates (Eds.), Handbook of teacher education: Globalization, standards, and professionalism in times of change (pp. 539-554). Dordrecht, The Netherlands: Springer.

Goddard, Y.L., Goddard, R.D., and Tschannen-Moran, M. (2007). A theoretical and empirical investigation of teacher collaboration for school improvement and student achievement in public elementary schools. Teachers College Record, 109(4), 877-896.

48 Ball, D.L. and Forzani, F.M. (2009). The work of teaching and challenge for teacher education. Journal of Teacher Education, 60(5), 497-511.