2018
Teaching and Learning International Survey
(TALIS 2018) Main
Study Recruitment and Field Test
Supporting Statement Part B
National Center for Education Statistics
Institute Of Education Sciences
U.S. Department Of Education
Washington, D.C.
May 2016
Revised July 2016
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
TALIS 2018 will acquire information from classroom teachers of International Standard Classification of Education (ISCED) Level 2 programs and the principles of their schools. ISCED Level 2 is also known as lower secondary education and usually lasts between 2 and 6 years, and begins around age 11. Middle school and junior high (grades 7 through 9) in the United States are classified as ISCED Level 2. The universe for the selection of schools is all schools with at least one ISCED Level 2 program in all states of the U.S. and the District of Columbia. Teachers that teach in special needs-only schools, that teach exclusively to adults, that serve as occasional or emergency teachers, or that are on long term leave are not within the scope of the study. Special education teachers working in regular public and private schools are eligible to participate. Additionally, very small schools and schools in remote areas may be considered ineligible for the study. For the field test, 35 schools, and for the main study, 200 schools will be selected to participate and, within each school, a random sample of up to 20 ISCED Level 2 teachers will be sampled (depending on the number of ISCED Level 2 faculty in each school), along with the school administrator, to participate. In schools with a small number of faculty (i.e., 20 or fewer teachers instructing at grades 7, 8, or 9), all teachers who teach at the target grade levels will be asked to participate. In schools with a large number of faculty (i.e., more than 20 teachers instructing at grades 7, 8, or 9), 20 teachers at the target grades will be randomly selected to participate.
The Technical Standards for TALIS 2018 field test and main study established by the international governing board include the following:
The teacher sample size must be a minimum of 600 ISCED Level 2 teachers for the field test and 4,000 ISCED Level 2 teachers for the main study, or the National Defined Target Population.
The school sample size must be a minimum of 30 schools for the field test and 200 schools for the main study, or all schools that have ISCED Level 2 teachers in the National Defined Target Population.
The target cluster size is 20 TALIS eligible teachers for the field study and for the main study.
School response rates must be at or above 75 percent of sampled schools. If a response rate is below 75 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. Although replacement schools could be called upon as substitutes for non-responding schools, national project managers (NPMs) will be encouraged to do all they can to obtain the participation of the schools in the original sample. As the number of replacement schools increases, the sample loses its probabilistic features and becomes increasingly “purposive.” This can undermine the reliability, validity, and interpretability of the country’s results. Responding schools that yield at least 50 percent of responding teachers will be considered to be participating schools; schools that fail to meet that threshold will be considered as “non-participating” even though the number of responding teachers may be enough to contribute to some of the analyses. Countries that experience less than 75 percent school participation after replacement have to demonstrate convincingly that their sample is not significantly biased. TALIS establishes three response rate zones—good, fair, or poor. “Good” means the country’s data will be included in the international sample. “Fair” means that the country’s data is a candidate for not being reported in international comparisons because the participation rate after replacement was less than 75 percent. However, in most cases, evidence was provided by the countries concerned that non responses bias was negligible. “Poor” means that the country’s data will not be included in the international comparisons. The TALIS Board of participating countries makes the final decision on whether to include the country’s data in international comparisons while taking into account various other factors.
The overall teacher response rates must be at or above 75 percent of sampled teachers.
Overview for TALIS 2018 field test and main study
The design for this study will be a stratified two-stage probability sampling design. This means that teachers (secondary stage units or secondary sampling units- SSU) will be randomly selected from the list of in-scope teachers in each of the randomly selected schools (first stage units, or primary sampling units- PSU).
The universe of interest is comprised of schools where ISCED Level 2 education is provided along with the affiliated principals and teachers. Following Indicators of Education Systems (INES) data collection definitions “the formal definition of a classroom teacher is a person whose professional activity involves the planning, organizing, and conduction of group activities whereby students’ knowledge, skills, and attitudes develop as stipulated by educational programs. In short, it is one whose main activity is teaching.” (OECD, 2004).
Target Populations
TALIS was designed to provide data on all ISCED Level 2 teachers in a participating country. TALIS identified policy issues that encompass the classroom, teacher, the school, and school management. No subject matter was excluded from the scope of TALIS. Thus, coverage of TALIS extends to all teachers of ISCED Level 2 and to the principals of the schools where they teach.
An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in programs at ISCED Level 2. In the U.S., ISCED Level 2 teachers are those who provide instruction at grades 7, 8, and/or 9. Teachers who teach a mixture of programs at different levels including ISCED Level 2 programs in the target school are included in the TALIS universe. There is no minimum cut-off for how much ISCED Level 2 teaching these teachers need to be engaged in. However, the international target population of TALIS restricts the survey to those teachers who teach regular classes in ordinary schools and to the principals of those schools. Teachers teaching only adults or in special needs-only schools are not part of the international target population and are deemed “out of scope.” Special education teachers working in regular public and private schools are eligible to participate.
When schools are comprised exclusively of these teachers, the school itself is said to be “out of scope.” Teacher aides, pedagogical support staff (e.g. guidance counselors, librarians), and health and social support staff (e.g. doctors, nurses, psychiatrists, psychologists, occupational therapists, and social workers) were not considered as teachers and thus not part of the TALIS international target population.
For national reasons, some participating countries will chose to restrict the coverage of their national implementation of TALIS to parts of the country. For example, a province or state experiencing civil unrest or an area struck by a natural disaster could be removed from the international target population to create a national target population. Participating countries were invited to keep these exclusions to a minimum.
Ideally, all the members of the target population ought to be eligible for sampling and data collection. This is the option that TALIS chose and, as a consequence, the international survey population (those who can be surveyed) is identical to the international target population (those who should be surveyed).
TALIS recognizes that attempting to survey teachers in very small schools, those in schools with no more than three teachers at ISCED Level 2, and those teaching in schools located in geographically remote areas could be a costly, time-consuming, and statistically inefficient exercise. Therefore, participating countries are allowed to exclude those teachers for TALIS data collection, thus creating a national survey population different from the national target population. The NPM is required to document the reasons for exclusion, the size, the location, the clientele, and so on for each excluded school. Moreover, during data collection in selected schools, the following teachers will be excluded: teachers who also act as principals (no teacher data collected, but principal data collected); substitute, emergency, or occasional teachers; teachers on long-term leave; and teachers teaching exclusively to adults. Ultimately, samples of schools and teachers will be selected from the national survey population.
Sampling Frame of Schools
The population of schools for TALIS 2018 is defined as all schools containing grades 7, 8, and/or 9. As in previous TALIS 2013, the school sampling frame will be developed from the most up-to-date NCES Common Core of Data (CCD) and Private Schools Survey (PSS) datasets. For the TALIS 2018 field test, we will use the school sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2017 which uses the 2014-2015 CCD and the 2013-2014 PSS school data. We will avoid, to the degree possible, schools sampled for NAEP, PISA, and TIMSS, all of which will also be collecting data in middle and high schools during the 2016-2017 school year.
Field Test Sampling
International standards do not require a formal probability sample of schools for the TALIS field test. It is sufficient that the samples of schools be representative of a broad range of schools from across the U.S. The field test requires a minimum teacher sample of 600 teachers. The U.S. plans to select a sample of 35 schools each with two substitute schools from across seven states (North Carolina, Pennsylvania, Arizona, Oregon, Michigan, Missouri, and West Virginia), with the expectation that 35 schools will ultimately participate, to provide for an adequate participating teacher sample. Among the 35 schools, 30 will be public schools and 5 will be private schools. This allows for school and teacher non-response and also for school level and within-school level exclusions.
Participating countries will provide Statistics Canada (the international sampling contractor) with a current and complete list of schools providing education at ISCED Level 2. This list will constitute the school sampling frame for TALIS and is expected to correspond to the survey population as defined and described on the Sampling Forms.
While the field test sample will be purposive by constraining the universe of schools to include only specific states, stratification for the U.S. field test will use a similar overall scheme as the main study, with the exception of the variables used as explicit strata. The stratification will use both explicit and implicit stratifying variables. Explicit strata for the field test will be defined by state and school control (public/private). Within each explicit stratum, schools will be sorted by grade structure (with three groups – middle-junior high with grades 6-8 or 7-9; high school with grades 9-12, or other schools with grade structures that include at least one ISCED 2 grade), locale (urban/suburban/town/rural), and percent minority students. An additional implicit stratifier is an estimate of and the number of ISCED 2 teachers used as the measure of size. Obtaining an accurate estimate of ISCED 2 teachers across schools is difficult and we plan to use the student enrollment by grade as an alternative variable for measure of size. Statistics Canada will select the sample using a systematic sampling method to select from the sorted list of schools. The probability of selection for each school will be proportional to the measure of size defined as the square root of the student enrollment by grade.
Teacher Samples
To allow for reliable estimation and modeling, while allowing for expected levels of non-response, the international required minimum sample size for the field test is set at 20 ISCED Level 2 teachers within each participating school. A minimum sample of 35 schools is to be drawn from the population of in-scope schools. Thus, the nominal international sample size for the field test is a minimum of 600 ISCED Level 2 teachers. We will plan to sample up to 20 teachers per school to meet the international requirement of at least 600 teachers in the field test sample.
To sample ISCED Level 2 teachers within each participating school, the international procedures call for the designation of a school coordinator in each school. In addition to assistance with arranging for the study within each school, the school coordinator of each school is asked to prepare a list of all eligible teachers in the school using a standardized Teacher Sampling Form (TSF). In the U.S., eligible teachers are those who conduct at least one class (in any subject matter) of 7th, 8th, and/or 9th graders. As long as a teacher conducts at least one class with students in this grade range, s/he is considered an ‘eligible teacher.’ Any teacher who does not teach any class at these grade levels is out of scope and will not be sampled for TALIS.
The TSF includes: identification number (number generated by sampling software); name of teacher; birth year; sex; main teaching domain (main subject taught); exclusion status (i.e., excluded or not); and email address. Each completed TSF is submitted to NCES (or its national contractor). This information is entered into the sampling software by Westat using WinW3S, a proprietary sampling software developed by the international contractor and used in other international studies such as TIMSS and PIRLS. The information collected from the TSF will remain with NCES and/or its national contractor. The names and email addresses of teachers are not associated with the TALIS database in any way; these are collected to allow for follow-up, should it become necessary. Once it is determined that no further follow-up is needed, all TSFs will be destroyed.
Main Study Sampling
For the TALIS 2018 main study, we will use the school sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2018 which will use the 2014-2015 CCD and the 2013-2014 PSS school data. The stratification will again use both explicit and implicit stratifying variables. Explicit strata for the main study will be defined by Census region (Northeast, Midwest, South, and West) and school control (public/private). Within each explicit stratum, schools will be sorted by grade structure (with three groups – middle-junior high with grades 6-8 or 7-9; high school with grades 9-12; or other schools with grade structures that include at least one ISCED 2 grade), locale (urban/suburban/town/rural), percent minority students, and number of ISCED 2 teachers (measure of school size).
For the main study, the international minimum number of teachers sampled is 4,000 teachers in 200 schools. As in the field test, the number of teachers sampled within schools in TALIS 2018 main study will be up to 20 teachers. Assuming the same response requirements as TALIS 2013, our initial target is a total sample of about 200 schools. To achieve the target final school response rate, we will use replacement schools to complete the sample. This larger sample size should allow for anticipated school refusals and to provide a large enough pool of eligible ISCED Level 2 teachers to obtain the minimum international teacher sample size of 4,000.
The international sampling specifications encourage countries to sample larger numbers of ISCED Level 2 schools and teachers than the minimum to ensure a sufficient sample size is achieved. The U.S. will implement the international sampling specifications as follows to achieve the required teacher sample size while allowing for anticipated non-response: schools with a small number of faculty at ISCED Level 2 (i.e., 20 or fewer), will have all eligible ISCED Level 2 teachers asked to participate; schools with a large number of faculty at ISCED Level 2 (i.e., more than 20), will have 20 ISCED Level 2 teachers randomly selected to participate. Thus, in some small-and medium-sized schools, less than 20 ISCED Level 2 teachers will be chosen, whereas in large-sized schools, 20 teachers will be chosen. Participating countries can suggest adaptations to the international sampling plan to better suit their national needs. The TALIS sampling team will review and approve all changes to the international sampling plan.
The same within-school sampling procedure used in the field test is used to generate a sample of teachers for the main study.
Nonresponse Bias Analysis, Weighting, Sampling Errors
It is inevitable that nonresponse will occur at both levels: school and teacher. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.
Our approach to maximizing school and teacher response rates in the field test includes the following:
Selecting and notifying schools as soon as OMB approval is received;
Approaching schools directly, and notifying states and school districts;
Assigning personal recruiters for specific schools;
Incentives for schools, school coordinators, and teachers (see Section A9);
Contact with schools and school coordinators at set intervals throughout the year preceding the survey; and
Collection of data in February 2017 (Field Test) and March-April 2018 (Main Study) to avoid conflicts with state testing which normally occurs in late spring/end of school year.
B.4 Purpose of Field Test and Main Study Data
Participation in the field test is an international requirement for participating in the TALIS 2018 main study. To prepare for the main study in 2018, TALIS countries will conduct the field test in the February of 2017, primarily to evaluate newly developed survey items but also to test the survey operations. The field test will include new teacher and school questionnaire items developed to address the TALIS 2018 framework. Finally, the field test will be used to test school recruitment, data collection, and data management procedures in preparation for the main study.
TALIS main study offers the opportunity for teachers and principals to provide input into education analysis and policy development. Cross-country analysis of this data will allow countries to identify other countries facing similar challenges and to learn from other policy approaches. School principals and teachers will provide information about issues such as the professional development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and recognition they receive about their work; and various other school leadership, management, and workplace issues.
The data from the TALIS main study will be reported via an international report that will be released in June 2019. NCES intends to produce a U.S. national report based on the TALIS main study data received from the international sponsors. The U.S. national report will be released at the same time as the international report from the OECD. This report will be tailored to a general audience and will report the U.S. results relative to other countries. There will not be a report on the field test results.
B.5 Individuals Consulted on Study Design
Many people at OECD, Statistics Canada, and other organizations around the world have been involved in the design of TALIS. Overall direction for TALIS in the U.S. is provided by Mary Coleman, the TALIS Governing Board representative and National Project Manager, and Patrick Gonzales, U.S. Department of Education.
Mary Coleman
National Center for Education Statistics
U.S. Department of Education
Patrick Gonzales
National Center for Education Statistics
U.S. Department of Education
References
OECD (2004), OECD Handbook for Internationally Comparative Education Statistics: Concepts, Standards, Definitions, and Classifications, OECD, Paris.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |