OECD
Teaching and Learning International Survey
(TALIS 2013 Field
Trial and Main Study)
Supporting Statement Part B
National Center for Education Statistics
Institute Of Education Sciences
U.S. Department Of Education
Washington, D.C.
August 1, 2011
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
TALIS 2013 will acquire information from classroom teachers of ISCED Level 2 programs and the principles of their schools. ISCED Level 2 is also known as lower secondary education and usually lasts between 2 and 6 years, and begins around age 11. Middle school and junior high (grades 7 through 9) in the United States are classified as ISCED Level 2. The universe for the selection of schools is all schools with at least one ISCED Level 2 program in all states of the United States and the District of Columbia. Teachers that teach in special needs-only schools, that teach exclusively to adults, that serve as occasional or emergency teachers, or that are on long term leave are not within the scope of the study. Special education teachers working in regular public and private schools are eligible to participate. Additionally, very small schools, schools in remote areas may be considered ineligible for the study. For the field trial, 20 schools will be selected to participate and within each school a random sample of 10 ISCED Level 2 teachers will be taken, and the school administrator will participate as well. For the main study, 275 schools will be selected to participate and within each school a sample of up to 30 ISCED Level 2 teachers will be chosen (depending on the number of ISCED Level 2 faculty in each school), along with the school administrator, to participate. In schools with a small number of faculty (i.e., 30 or fewer teachers instructing at grades 7, 8 or 9), all teachers who teach at the target grade levels will be asked to participate. In schools with a large number of faculty (i.e., more than 30 teachers instructing at grades 7, 8, or 9), 30 teachers at the target grades will be randomly selected to participate.
The Technical Standards for TALIS 2013 field trial and main study established by the international governing board include the following:
The teacher sample size must be a minimum of 200 surveyed ISCED Level 2 teachers for the field trial and 4,000 surveyed ISCED Level 2 teachers for the main study, or the National Defined Target Population.
The school sample size must be a minimum of 20 schools for the field trial and 200 schools for the main study, or all schools that have ISCED Level 2 teachers in the National Defined Target Population.
The target cluster size is typically 10 TALIS eligible teachers for the field study and 20 TALIS eligible teachers for the main study.
School response rates must be above 75 percent of sampled schools. If a response rate is below 75 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. Although replacement schools could be called upon as substitutes for non-responding schools, national project managers (NPMs) will be encouraged to do all they can to obtain the participation of the schools in the original sample. As the number of replacement schools increases, the sample loses its probabilistic features and becomes increasingly “purposive”. This can undermine the reliability, validity, and interpretability of the country’s results. Responding schools that yield at least 50 percent of responding teachers will be considered as participating schools; schools that fail to meet that threshold will be considered as “non participating” even though the number of responding teachers may be enough to contribute to some of the analyses. Countries that experience less than 75 percent school participation after replacement have to demonstrate convincingly that their sample is not significantly biased. (For a description of the steps required by NCES Standards for surveys that do not meet established participation rates, see Supporting Document E.) TALIS establishes three response rate zones—good, fair, or poor. “Good” means the country’s data will be included in the international sample. “Fair” means that the country’s data is a candidate for not being reported in international comparisons because the participation rate after replacement was less than 75 percent. However, in most cases, evidence was provided by the countries concerned that non responses bias was negligible. “Poor” means that the country’s data will not be included in the international comparisons. The TALIS Board of participating countries makes the final decision on whether to include the country’s data in international comparisons while taking into account various other factors.
The overall teacher response rates must be 75 percent of sampled teachers.
Overview for TALIS 2013 field trial and main study
The design for this study will be a stratified two-stage probability sampling design. This means that teachers (secondary stage units or secondary sampling units- SSU) will be randomly selected from the list of in-scope teachers in each of the randomly selected schools (first stage units, or primary sampling units- PSU).
The universe of interest is comprised of schools where ISCED Level 2 education is provided along with the affiliated principals and teachers. Following Indicators of Education Systems (INES) data collection definitions “the formal definition of a classroom teacher is a person whose professional activity involves the planning, organizing, and conduction of group activities whereby students’ knowledge, skills, and attitudes develop as stipulated by educational programs. In short, it is one whose main activity is teaching.” (OECD, 2004).
Target Populations
TALIS was designed to cover all ISCED Level 2 teachers in a participating country. TALIS identified policy issues that encompass the classroom, teacher, the school, and school management. No subject matter was excluded from the scope of TALIS. Thus, coverage of TALIS extends to all teachers of ISCED Level 2 and to the principals of the schools where they teach.
An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in programs at ISCED Level 2. In the United States, ISCED Level 2 teachers are those who provide instruction at grades 7, 8, and/or 9. Teachers who teach a mixture of programs at different levels including ISCED Level 2 programs in the target school are included in the TALIS universe. There is no minimum cut-off for how much ISCED Level 2 teaching these teachers need to be engaged in.
The international target population of TALIS restricts the survey to those teachers who teach regular classes in ordinary schools and to the principals of those schools. Teachers teaching to adults and teachers that teach in special needs-only schools are not part of the international target population and are deemed “out of scope.” Special education teachers working in regular public and private schools are eligible to participate.
When schools are comprised exclusively of these teachers, the school itself is said to be “out of scope.” Teacher aides, pedagogical support staff (e.g. guidance counselors, librarians), and health and social support staff (e.g. doctors, nurses, psychiatrists, psychologists, occupational therapists, and social workers) were not considered as teachers and thus not part of the TALIS international target population.
For national reasons, some participating countries will chose to restrict the coverage of their national implementation of TALIS to parts of the country. For example, a province or state experiencing civil unrest or an area struck by a natural disaster could be removed from the international target population to create a national target population. Participating countries were invited to keep these exclusions to a minimum.
Ideally, all the members of the target population ought to be eligible for sampling and data collection. This is the option that TALIS chose and, as a consequence, the international survey population (those who can be surveyed) is identical to the international target population (those who should be surveyed).
TALIS recognizes that attempting to survey teachers in very small schools, those in schools with no more than three teachers at ISCED Level 2, and those teaching in schools located in geographically remote areas could be a costly, time-consuming, and statistically inefficient exercise. Therefore, participating countries are allowed to exclude those teachers for TALIS data collection, thus creating a national survey population different from the national target population. The NPM is required to document the reasons for exclusion, the size, the location, the clientele, and so on for each excluded school. Moreover, as discussed later in this section, during data collection in the selected schools, some teachers could be excused from data collection.
Ultimately, samples of schools and teachers will be selected from the national survey population.
Within a selected in-scope school, the following teachers are to be excluded: teachers who also act as principals (no teacher data collected, but principal data collected); substitute, emergency, or occasional teachers; teachers on long-term leave; and teachers teaching exclusively to adults.
Sampling Frame of Schools
Participating countries will provide Statistics Canada (the international sampling contractor) with a current and complete list of schools providing education at ISCED Level 2. This list constitutes the school sampling frame for TALIS and is expected to correspond to the survey population as defined and described on the Sampling Forms. The sampling frame will be derived from the most recent year available of the Common Core of Data (CCD) and Private School Survey (PSS) datasets. For the field trial, the United States will use the 2009-2010 CCD and PSS datasets; for the main study, the United States will use the 2011-2012 CCD and PSS datasets.
The sampling frame will contain certain key variables: a national school identifier, a measure of size (MOS), preferably the number of ISCED Level 2 teachers, and values for those variables to be used for stratification; whenever possible, the type of funding (private or public) and the type of education stream (academic or vocational) will also to appear on the frame. Additional sampling frames are required for the sampling of teachers, namely, the list of eligible ISCED Level 2 teachers in each selected school.
The sampling frame for the main study will use two explicit strata: public/private and urbanicity (i.e., urban, suburban, rural). All other variables will be implicit stratification variables.
The United States will select a sample of 275 schools that provide education at grades 7, 8, and/or 9. This is a larger number of schools than the minimum required by the international sampling specifications, but was chosen to allow for anticipated school refusals and to provide a large enough pool of eligible ISCED Level 2 teachers to obtain the minimum international teacher sample size of 4,000.
Teacher Samples
To allow for reliable estimation and modeling, while allowing for expected levels of non-response, the minimum sample size for the field trial is set at 10 ISCED Level 2 teachers within each participating school. A minimum sample of 20 schools is to be drawn from the population of in-scope schools. Thus, the nominal international sample size for the field trial is a minimum of 200 ISCED Level 2 teachers. For the main study, the minimum sample size is set at 20 ISCED Level 2 teachers within each participating school and a minimum sample of 200 schools to be drawn from the population of in-scope schools with the nominal international sample size a minimum of 4,000 teachers. However, the international sampling specifications encourage countries to sample larger numbers of ISCED Level 2 schools and teachers than the minimum to ensure a sufficient sample size is achieved. The United States will implement the international sampling specifications as follows to achieve the required teacher sample size while allowing for anticipated non-response: schools with a small number of faculty at ISCED Level 2 (i.e., 30 or fewer), will have all eligible ISCED Level 2 teachers asked to participate; schools with a large number of faculty at ISCED Level 2 (i.e., more than 30), will have 30 ISCED Level 2 teachers randomly selected to participate. Thus, in some small- and medium-sized schools, less than 30 ISCED Level 2 teachers will be chosen, whereas in large-sized schools, 30 teachers will be chosen. Participating countries can choose to augment their national sample by selecting more schools, or by selecting more teachers within each selected school, or by increasing both. The sample size requirement will be reduced for some participating countries because of the smaller number of schools available for sampling. In a few cases, because the average number of teachers in the schools will be less than expected in the international plan, the number of schools to be sampled can be increased to maintain a minimum total number of participating teachers.
Participating countries can suggest variations or adaptations of the international sampling plan to better suit their national needs. The TALIS sampling team will review and approve all changes to the international sampling plan.
To sample ISCED Level 2 teachers within each sampled school, the international procedures call for the designation of a school coordinator in each school. In addition to assistance with arranging for the study within each school, the school coordinator of each school is asked to prepare a list of all teachers of all eligible teachers in the school (using a standardized Teacher Sampling Form (TSF)). In the United States, eligible teachers are those teachers who conduct at least one class (subject matter not limited) of 7th, 8th, and/or 9th graders. As long as a teacher conducts at least one class with students in this grade range, s/he is considered an ‘eligible teacher.’ Any teacher who does not teach any class at these grade levels is out of scope and will not be sampled for TALIS.
The TSF includes: identification number (number generated by sampling software); name of teacher; birth year; sex; main teaching domain (main subject taught); exclusion status (i.e., excluded or not); and email address. Each completed TSF is submitted to NCES (or its national contractor). This information is entered into the sampling software by NCES or its national contractor (WinW3S, a proprietary sampling software developed by the international contractor and used in other international studies such as TIMSS and PIRLS). Based on the list of teachers as well as the total number of eligible ISCED Level 2 teachers in each school, teachers will be selected to participate. The information collected from the TSF remains with NCES and/or its national contractor. The names and email addresses of teachers are not associated with the TALIS database in any way; these are collected to allow for follow-up, should it be necessary. Once it is determined that no further follow-up is needed, all TSFs will be destroyed.
Nonresponse Bias Analysis, Weighting, Sampling Errors
In order to meet international standards for participation, for the main study the United States must achieve minimum response rates established by OECD and also meet OMB and NCES standards. While the OECD has set a minimum response rate of 75 percent of sampled schools (after replacement), NCES’s response rate standards apply to each data collection unit (in this case, schools, and teachers) and are calculated WITHOUT replacements. NCES Statistical Standard 2-2-2 indicates that for a cross-sectional survey (without an assessment), the target response rate is at least 85 percent for each sampling unit (see http://nces.ed.gov/statprog/2002/std2_2.asp).
At the teacher level, the OECD has set a minimum response rate of 75 percent of selected teachers (in both original and replacement schools). Participating schools that fail to yield at least a 50 percent response rate among the sampled teachers will be considered as non-participating, even though the number of participating teachers may be enough to contribute to some of the analyses. NCES standards require a minimum response rate of 85 percent of sampled teachers.
It is inevitable that nonresponse will occur at both levels: school and teacher. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, at minimum complying with OMB and NCES Statistical Standards. According to the standards, a nonresponse bias analysis (NRBA) must be conducted if either the school or teacher response rate falls below this threshold (main study only). After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification. Although the NRBA will include a comparison of frame variables for respondents and nonrespondents, the specific nature of the analysis and how extensive that analysis is will be a function of the amount of nonresponse combined with the results from the initial frame analysis.
As part of the NRBA, two forms of analysis are usually undertaken: a test of the independence of each participating individual’s characteristic and participation status, and a logistic regression in which the conditional independence of selected individual characteristics as predictors of participation is examined. This analysis is required by NCES to support a decision to include U.S. results in the international report in the event the response rate is below 85 percent. The report may also be required by OECD should U.S. response rates fall below the minimum of 75 percent of schools or teachers. An example of a recent non-response bias analysis can be found in chapter 3, section 3.4 of the TIMSS 2007 Technical Report at http://nces.ed.gov/pubs2009/2009012.pdf.
In the event that recruitment efforts indicate that response rates will likely be below the minimum threshold set by NCES, the contractor is required to submit an outline for the non-response bias analysis report 1 month prior to data collection, or as soon as it becomes clear that a non-response bias analysis will be needed. A preliminary report will be due within 2 months of the completion of data collection so that results of the non-response bias analysis can be used in pre-release decisions about U.S. sample coverage. The contractor shall produce weights to be used in this analysis because weights produced by the International Consortium will not be available until a later date. An updated version of the non-response bias analysis report must be produced after weights are provided by the International Consortium. This report must be reviewed and approved by the NCES Chief Statistician. A reduced form of the final report (a summary of findings), with final data and weights provided by the International Consortium, also must be included in the national report (to be included in the report’s technical appendix). The complete, approved non-response bias analysis report will be included in the data file user’s guide.
Our approach to maximizing school and teacher response rates in the field trial includes the following:
Selecting and notifying schools as soon as OMB approval is received;
Approaching schools directly, and notifying states and school districts;
Assigning personal recruiters for specific schools;
Incentives for schools, school coordinators, and teachers (see Section A9);
Contact with schools and school coordinators at set intervals throughout the year preceding the survey; and
Collection of data in March-April of 2012 (Field Trial) and March-April 2013 (Main Study) to avoid conflicts with state testing which normally occurs in late spring/end of school year.
B.4 Purpose of Field Trial and Main Study Data Uses
The Teaching and Learning International Survey (TALIS) offers the opportunity for teachers and principals to provide input into education analysis and policy development. TALIS is being conducted by the Organization for Economic Cooperation and Development (OECD). Cross-country analysis of this data will allow countries to identify other countries facing similar challenges and to learn from other policy approaches. School principals and teachers will provide information about issues such as the professional development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and recognition they receive about their work; and various other school leadership, management, and workplace issues. In the TALIS study, it is our intention to draw a picture of the different educational practices in all the participating countries. Countries and individuals may differ in their educational approaches.
The data from the TALIS main study will be reported via an international report that will be released in June 2014. NCES intends to produce a U.S. national report based on the TALIS main study data received from the international sponsors. The U.S. national report will be released at the same time as the international report from the OECD. This report will be tailored to a general audience and will report results of the United States relative to other countries. There will not be a report on the field trial results.
B.5 Individuals Consulted on Study Design
Many people at OECD, Statistics Canada, and other organizations around the world have been involved in the design of TALIS. Overall direction for TALIS in the U.S. is provided by Patrick Gonzales, the TALIS Board of Participating Countries representative, and Erin Roth, the TALIS National Project Manager at the National Center for Education Statistics, U.S. Department of Education.
Patrick Gonzales
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
415-920-9229
Erin Roth
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
202-502-7415
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |