Part B TALIS 2018 Main Study

2018 Teaching and Learning International Survey
(TALIS 2018) Main Study

OMB# 1850-0888 v.7

Supporting Statement Part B

National Center for Education Statistics

Institute Of Education Sciences

U.S. Department Of Education

Washington, D.C.

November 2017

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe

TALIS 2018 will acquire information from classroom teachers of International Standard Classification of Education (ISCED) Level 2 programs and the principles of their schools. ISCED Level 2 is also known as lower secondary education and usually lasts between 2 and 6 years, and begins around age 11. Middle school and junior high (grades 7 through 9) in the United States are classified as ISCED Level 2. The universe for the selection of schools is all schools with at least one ISCED Level 2 program in all states of the U.S. and the District of Columbia. Teachers that teach in special needs-only schools, that teach exclusively to adults, that serve as occasional or emergency teachers, or that are on long term leave are not within the scope of the study. Special education teachers working in regular public and private schools are eligible to participate. Additionally, very small schools and schools in remote areas may be considered ineligible for the study. For the main study, 220 schools will be selected to participate and, within each school, a random sample of up to 20 ISCED Level 2 teachers will be sampled (depending on the number of ISCED Level 2 faculty in each school), along with the school administrator, to participate. In schools with a small number of faculty (i.e., 20 or fewer teachers instructing at grades 7, 8, or 9), all teachers who teach at the target grade levels will be asked to participate. In schools with a large number of faculty (i.e., more than 20 teachers instructing at grades 7, 8, or 9), 20 teachers at the target grades will be randomly selected to participate.

B.2 Procedures for the Collection of Information

This section presents information on the TALIS international standards and description for school and teacher sampling, recruitment, and data collection procedures for TALIS 2018 main study. Gaining schools’ and teachers’ cooperation in voluntary research is increasingly challenging and employing effective strategies for gaining the cooperation of schools is central to the data collection effort. Main study states, districts, and schools will be recruited beginning in early fall 2017 (OMB approval # 1850-0888 v.4-6). The main study data collection will be conducted from March 1 through April 27, 2018.

B.2a Statistical Methodology

The Technical Standards for TALIS 2018 main study established by the international governing board include the following:

• The teacher sample size must be a minimum of 4,000 ISCED Level 2 teachers for the main study, or the National Defined Target Population.

• The school sample size must be a minimum 200 schools for the main study, or all schools that have ISCED Level 2 teachers in the National Defined Target Population.

• The target cluster size is 20 TALIS eligible teachers for the main study.

• School response rates must be at or above 75 percent of initially sampled schools. If a response rate is below 75 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. Although replacement schools could be called upon as substitutes for non-responding schools, national project managers (NPMs) will be encouraged to do all they can to obtain the participation of the schools in the original sample. As the number of replacement schools increases, the sample loses its probabilistic features and becomes increasingly “purposive.” This can undermine the reliability, validity, and interpretability of the country’s results. Responding schools that yield at least 50 percent of responding teachers will be considered to be participating schools; schools that fail to meet that threshold will be considered as “non-participating” even though the number of responding teachers may be enough to contribute to some of the analyses. Countries that experience less than 75 percent school participation after replacement have to demonstrate convincingly that their sample is not significantly biased. TALIS establishes three response rate zones—good, fair, or poor. “Good” means the country’s data will be included in the international sample. “Fair” means that the country’s data is a candidate for not being reported in international comparisons because the participation rate after replacement was less than 75 percent. However, in most cases, evidence was provided by the countries concerned that non responses bias was negligible. “Poor” means that the country’s data will not be included in the international comparisons. The TALIS Board of participating countries makes the final decision on whether to include the country’s data in international comparisons while taking into account various other factors.

• The overall teacher response rates must be at or above 75 percent of sampled teachers.

Overview for TALIS 2018 Main Study Sample

The design for this study is a stratified two-stage probability sampling design. This means that teachers (secondary stage units or secondary sampling units- SSU) will be randomly selected from the list of in-scope teachers in each of the randomly selected schools (first stage units, or primary sampling units- PSU).

The universe of interest is comprised of schools where ISCED Level 2 education is provided along with the affiliated principals and teachers. Following Indicators of Education Systems (INES) data collection definitions “the formal definition of a classroom teacher is a person whose professional activity involves the planning, organizing, and conduction of group activities whereby students’ knowledge, skills, and attitudes develop as stipulated by educational programs. In short, it is one whose main activity is teaching.” (OECD, 2004).

Target Populations

TALIS was designed to provide data on all ISCED Level 2 teachers in a participating country. TALIS identified policy issues that encompass the classroom, teacher, the school, and school management. No subject matter was excluded from the scope of TALIS. Thus, coverage of TALIS extends to all teachers of ISCED Level 2 and to the principals of the schools where they teach.

An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in programs at ISCED Level 2. In the U.S., ISCED Level 2 teachers are those who provide instruction at grades 7, 8, and/or 9. Teachers who teach a mixture of programs at different levels including ISCED Level 2 programs in the target school are included in the TALIS universe. There is no minimum cut-off for how much ISCED Level 2 teaching these teachers need to be engaged in. However, the international target population of TALIS restricts the survey to those teachers who teach regular classes in ordinary schools and to the principals of those schools. Teachers teaching only adults or in special needs-only schools are not part of the international target population and are deemed “out of scope.” Special education teachers working in regular public and private schools are eligible to participate. The national TALIS target population is the same as the international target population, with all the members of the target population to be eligible for sampling and data collection.

When schools are comprised exclusively of these teachers, the school itself is said to be “out of scope.” Teacher aides, pedagogical support staff (e.g. guidance counselors, librarians), and health and social support staff (e.g. doctors, nurses, psychiatrists, psychologists, occupational therapists, and social workers) were not considered as teachers and thus not part of the TALIS international target population.

During data collection in selected schools, the following teachers will be excluded: teachers who also act as principals (no teacher data collected, but principal data collected); substitute, emergency, or occasional teachers; teachers on long-term leave; and teachers teaching exclusively to adults. Ultimately, samples of schools and teachers will be selected from the national survey population.

Sampling Frame of Schools

The population of schools for TALIS 2018 is defined as all schools containing grades 7, 8, and/or 9. As in previous TALIS 2013, the school sampling frame was developed from the most up-to-date NCES Common Core of Data (CCD) and Private Schools Survey (PSS) datasets. The school frame for the TALIS 2018 main study national frame was developed from the 2018 National Assessment of Educational Progress (NAEP) school frame, for which the sampling frame for public schools was based on the 2015-2016 CCD and for private schools on the 2015-2016 PSS school data. We avoided, to the degree possible, schools sampled for the 2017-18 National Teacher and Principal Survey (NTPS) and the 2018 Program for International Student Assessment (PISA), both of which will also be collecting data in middle and high schools during the 2017-2018 school year.

Field Test School Sampling

International standards did not require a formal probability sample of schools for the TALIS field test. It was sufficient that the samples of schools be representative of a broad range of schools from across the U.S. The field test required a minimum teacher sample of 600 teachers. The U.S. selected a sample of 70 original schools each with one replacement in order to ensure that there were enough schools to gain participation from the required 35 schools. The schools were chosen from across seven states (North Carolina, Pennsylvania, Arizona, Oregon, Michigan, Missouri, and West Virginia), with the expectation that 35 schools would ultimately participate, to provide for an adequate participating teacher sample. Among the 35 schools, 30 were public schools and 5 were private schools. This allowed for school and teacher non-response and also for school level and within-school level exclusions.

Participating countries provided Statistics Canada (the international sampling contractor) with a current and complete list of schools providing education at ISCED Level 2. This list constituted the school sampling frame for TALIS and was expected to correspond to the survey population as defined and described on the study sampling forms.

While the field test sample was purposive by constraining the universe of schools to include only specific states, stratification for the U.S. field test used a similar overall scheme as the main study, with the exception of the variables used as explicit strata. The stratification used both explicit and implicit stratifying variables. Explicit strata for the field test were defined by state and school control (public/private). Within each explicit stratum, schools were sorted by grade structure (with three groups – middle-junior high with grades 6-8 or 7-9; high school with grades 9-12, or other schools with grade structures that include at least one ISCED 2 grade), locale (urban/suburban/town/rural), and percent minority students. An additional implicit stratifier was an estimate of the number of ISCED 2 teachers used as the measure of size. Obtaining an accurate estimate of ISCED 2 teachers across schools is difficult and we used the student enrollment by grade as an alternative variable for measure of size. Statistics Canada selected the sample using a systematic sampling method to select from the sorted list of schools. The probability of selection for each school was proportional to the measure of size defined as the square root of the student enrollment by grade.

Main Study School Sampling

For the TALIS 2018 main study, we used the school sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2018 which was based on the 2015-2016 CCD and the 2015-2016 PSS school data. The stratification again used both explicit and implicit stratifying variables. Explicit strata for the main study were defined by school control (public/private) and grade structure (with three groups: middle/junior high schools with grades 6-8 or 7-9, high schools with grades 9-12, and other schools with grade structures that include at least one ISCED 2 grade). Within each explicit stratum, schools were sorted by census region (Northeast, Midwest, South, and West), locale (urban/suburban/town-rural), percent minority students, state, and number of ISCED 2 students (measure of school size)¹.

For the main study, the international minimum number of teachers sampled is 4,000 teachers in 200 schools. As in the field test, the number of teachers sampled within schools in TALIS 2018 main study will be up to 20 teachers. Assuming the same response requirements as TALIS 2013, our initial target is a total sample of about 200 schools. To achieve the target final school response rate, we will use replacement schools to complete the sample. This larger sample size should allow for anticipated school refusals and to provide a large enough pool of eligible ISCED Level 2 teachers to obtain the minimum international teacher sample size of 4,000.

The international sampling specifications encourage countries to sample larger numbers of ISCED Level 2 schools and teachers than the minimum to ensure a sufficient sample size is achieved. The U.S. will implement the international sampling specifications as follows to achieve the required teacher sample size while allowing for anticipated non-response: schools with a small number of faculty at ISCED Level 2 (i.e., 20 or fewer) will have all eligible ISCED Level 2 teachers asked to participate, and schools with a large number of faculty at ISCED Level 2 (i.e., more than 20) will have 20 ISCED Level 2 teachers randomly selected to participate. Thus, in some small-and medium-sized schools, less than 20 ISCED Level 2 teachers will be chosen, whereas in large-sized schools, 20 teachers will be chosen. Participating countries can suggest adaptations to the international sampling plan to better suit their national needs. The TALIS sampling team reviewed and approved all changes to the international sampling plan.

Field Test Teacher Sampling

To allow for reliable estimation and modeling, while allowing for expected levels of non-response, the international required minimum sample size for the field test was set at 20 ISCED Level 2 teachers within each participating school. A minimum sample of 70 schools was drawn from the population of in-scope schools. Once we achieved the minimum necessary number of 35 participating schools, we ceased recruitment. The nominal international sample size for the field test was a minimum of 600 ISCED Level 2 teachers. We sampled up to 20 teachers per school to meet the international requirement of at least 600 teachers in the field test sample.

To sample ISCED Level 2 teachers within each participating school, the international procedures called for the designation of a school coordinator in each school. In addition to assistance with arranging for the study within each school, the school coordinator of each school was asked to prepare a list of all eligible teachers in the school using a standardized Teacher Sampling Form (TSF). In the U.S., eligible teachers are those who conduct at least one class (in any subject matter) of 7^th, 8^th, and/or 9^th graders. As long as a teacher conducts at least one class with students in this grade range, s/he is considered an ‘eligible teacher.’ Any teacher who does not teach any class at these grade levels is out of scope and is not sampled for TALIS.

The TSF included: identification number (number generated by sampling software); name of teacher; main teaching domain (main subject taught); exclusion status (i.e., excluded or not); and email address. Each completed TSF was submitted to Westat (the national contractor administering TALIS in the U.S.). This information was entered into the sampling software by Westat using WinW3S, a proprietary sampling software developed by the international contractor and used in other international studies such as TIMSS and the Progress in International Reading Literacy Study (PIRLS). The information collected from the TSF remained with Westat. The names and email addresses of teachers were not associated with the TALIS database in any way. They were collected to allow direct communication with sampled teachers to provide them with the survey link and access information, and for follow-up communications. Once it was determined that no further follow-up was needed, all TSFs were destroyed.

Main Study Teacher Sampling

For the main study, up to 20 teachers will be sampled within each participating school in order to yield the minimum required 4,000 teacher national sample.

As in the field test, each school will have a school coordinator to provide assistance with arranging for the study and prepare a list of all eligible teachers in the school using a standardized Teacher Sampling Form (TSF). In the U.S., eligible teachers are those who conduct at least one class (in any subject matter) of 7^th, 8^th, and/or 9^th graders. As long as a teacher conducts at least one class with students in this grade range, s/he is considered an ‘eligible teacher.’ Any teacher who does not teach any class at these grade levels is out of scope and is not sampled for TALIS.

The TSF is unchanged from the one used in the field test. Each completed TSF will be submitted to Westat and the information entered into the sampling software using WinW3S. The information collected from the TSF will only be used to select the teacher sample in each school and to establish direct communication. Once no further follow-up with sampled teachers is necessary, all TSF information will be destroyed.

Nonresponse Bias Analysis, Weighting, Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and teacher. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.

B.2.b Respondent Recruitment

Based on recruitment knowledge gained in the TALIS 2018 field trial and other NCES studies, states and districts will be notified about the study before the sampled schools in their jurisdictions are contacted about the study. Before agreeing to participate in the study, school staff often want to be assured that their school district knows about the study. Our proposed approach for state, district, and school recruitment is described in this section. See Appendix A and B for all of the respondent materials referenced below.

State Recruitment. In fall 2017, state education agencies (SEAs) in states that contain TALIS 2018 sampled main study schools will be mailed a package that includes the state letter and TALIS 2018 advanced materials. We will work with the NAEP State Coordinators (NSCs) in the state recruitment effort. Some NSCs may wish to send the state letter themselves, as well as follow-up to answer any questions; otherwise, Westat school recruiters will mail the state package.

District Recruitment. In fall 2017, an advance package to district superintendents will be mailed. The package will contain an introductory letter, including a list of sampled schools in their jurisdiction, and the TALIS 2018 advanced materials. The district mailings will come from the NSC or Westat, depending on each NSC’s preference. Shortly after the mailing, the district superintendent will be contacted by phone to inform them of the study, ensure they received the TALIS 2018 package, and answer any questions they may have. Any issues with approaching schools in the district will also be discussed at the time.

The TALIS 2018 study staff will respond to districts’ requirements such as research applications or meetings to provide more information about the study. If a district chooses not to participate, the recruiter will document all concerns listed by the district so that a strategy can be formulated for refusal conversion attempts.

Based on experience from the TALIS 2018 field test and previous NCES studies, some NSCs will want to talk to district staff themselves, while others will mail the package but do no further contact districts, and still others will not want to be involved in district recruitment at all. In cases where the NSCs do not wish to follow-up, Westat’s recruiters will work directly with the districts.

Some districts will be designated as “special handling districts”, based on prior recruitment experience on a variety of NCES studies. Contacting special handling districts will begin with updating district information based on what can be gleaned from online sources, followed by calls to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiry will also be made about the amount of time the district spends reviewing similar research applications.

School Recruitment. After each district of sampled public schools has been informed of the study and has confirmed the receipt of the TALIS 2018 package, the main study public school mailings will be triggered. Private school mailings will begin simultaneously with public districts being contacted. All of the school mailings (expected to take place throughout fall 2017), will contain an introductory letter, offering each sampled school a $200 check as a thank you for participation (to be mailed with a thank you letter after the end of the data collection), the TALIS brochure, and school administrator and teacher Frequently Asked Questions (FAQs) sheets. Shortly after the mailing, the NSCs or Westat recruiters will phone the school administrator to: discuss the study, gain cooperation, and assign a school staff person to serve as the TALIS 2018 school coordinator who will work with the TALIS staff to manage the data collection in the school. The school coordinator will act as the liaison between study staff and their school.

In cases where recruitment proves more difficult, school recruiters will consult with the TALIS home office to evaluate a conversion plan for each school. Typically, the types of general issues faced are principals who are difficult to reach, school staff who are considering participation but not providing a final decision, principals and/or staff who express a concern and need follow-up, and principals who may require additional appreciation for agreeing to participate.

School Coordinator Contact. Shortly after permission has been granted from the school administrator, Westat will also email the school coordinator MyTALIS website registration information. The MyTALIS website (described in more detail in the next section) will contain the e-filing instructions and templates, and other survey logistical information. Each school will receive a unique ID that will support multiple school users. Each user must, however, provide their own contact information and setup their own unique account. This registration process has been used across all of NCES’ international studies and in NAEP.

In January 2018, the school coordinator will receive a handbook detailing the procedures for administering the TALIS 2018 survey in the school, and providing timelines and instructions for submission of the list of teachers via MyTALIS. Westat TALIS 2018 staff will also call the school coordinator to discuss the TALIS activities at the school, including when to begin constructing and e-filing the teacher list.

Teacher Recruitment. As described above, teacher lists, including teachers’ names and email addresses will be collected in early 2018 for drawing a teacher sample in each school. Edit checks will be run during sampling, checking column mappings and completeness of data to ensure that all teacher listing forms are constructed properly for sampling and for contacting the sampled teachers (see Appendix A).

After sampling has been completed and the correct teacher names and email addresses have been collected for a school, the school coordinator will be mailed a package containing a set of personalized teacher and principal invitations to the TALIS survey with the survey address and credentials and the teacher FAQ sheet. The school coordinator will be asked to distribute these materials to the appropriate teachers, to serve as the study’s main point of contact for the teachers, and to answer any questions from the school administrator and teachers about TALIS 2018. Shortly after sampling, Westat will also email each sampled teacher a direct, unique link to the teacher questionnaire.

Role of the MyTALIS Website. The central purpose of MyTALIS is to provide a way for the schools to securely upload a list of teachers and to track participation of sampled teachers during the survey. MyTALIS is also used as a source for information about TALIS such as copies of advanced materials and descriptions of survey activities, school actions, and instructions and templates for submitting lists. School registrants are encouraged to update their contact information and access information about the study in MyTALIS.

B.2.c TALIS Data Collection

Data collection will occur in March-April 2018 for the main study. Prior to data collection, Westat TALIS staff will complete various pre-survey activities.

Pre-survey Activities

Beginning in February, school coordinators will be asked to do the following activities:

• Create and submit a list of eligible teachers via MyTALIS

• Receive and distribute study materials from the TALIS Home Office

• Encourage sampled teaches and the principal to complete their questionnaire

As discussed, in January 2018, the school coordinators will receive a handbook and instructions for assembling a teacher list. The lists will be submitted to Westat via MyTALIS and the teacher samples will be drawn during February 2018. Eligible teachers are those that teach each grade 7, 8, or 9 students. Teacher-aides, assistants, and other school staff are not eligible. The sampled teachers will be uploaded to MyTALIS in a tracking table containing the teacher TALIS study ID, the status of the teacher questionnaire and principal questionnaire, and the teacher participation status. The latter is set to a default status and changed by the school coordinator as appropriate (see below).

Data Collection

The principal and teacher questionnaire will be available electronically, with a hard copy available upon request. The main study TALIS questionnaires are provided in Appendix B. The principal and teachers will be given links to their questionnaires as part of invitation to TALIS and will also be emailed their link to the questionnaire.

Throughout the survey data collection period, TALIS staff and the school coordinator will monitor the return of questionnaires and, working in conjunction, will follow-up with non-responders as needed. A ‘window-is-closing’ non-response follow-up effort will be utilized, gently reminding respondents to complete their questionnaires. The two reminders before the last, near the end of data collection, are designed to establish a deadline effect, and will be followed with an extension email. This campaign-style approach is designed to provide soft reminders across the data collection window, while creating a sense of urgency to respond.

School and school coordinator incentive checks will be distributed after the principal questionnaire is completed. Teacher incentive checks will be distributed as the teacher questionnaires are completed. Incentives will be mailed to schools on a weekly basis throughout the data collection period. The school coordinators will distribute the incentive checks.

B.3 Maximizing Response Rates

Our approach to maximizing school and teacher response rates in the main study includes the following:

• Communicating with state and district officials early in the process and applying a more proactive approach with states by coordinating with NAEP State Coordinators to gain assistance with sampled schools;

• Contacting schools as soon as OMB approval is received;

• Assigning personal recruiters for specific schools;

• Incentives for schools, school coordinators, and teachers (see Section A9);

• Contact with schools and school coordinators at set intervals throughout the year preceding the survey; and

• Collection of data in March-April 2018 to avoid conflicts with state testing which normally occurs in late spring/end of school year.

B.4 Purpose of Field Test and Main Study Data

Participation in the field test is an international requirement for participating in the TALIS 2018 main study. To prepare for the main study in 2018, TALIS countries conducted the field test in the February of 2017, primarily to evaluate newly developed survey items but also to test the survey operations. The field test included new teacher and school questionnaire items developed to address the TALIS 2018 framework. Finally, the field test was used to test school recruitment, data collection, and data management procedures in preparation for the main study.

TALIS main study offers the opportunity for teachers and principals to provide input into education analysis and policy development. Cross-country analysis of this data will allow countries to identify other countries facing similar challenges and to learn from other policy approaches. School principals and teachers will provide information about issues such as the professional development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and recognition they receive about their work; and various other school leadership, management, and workplace issues.

The data from the TALIS main study will be reported via an international report that will be released in June 2019. NCES intends to produce a U.S. national report based on the TALIS main study data received from the international sponsors. The U.S. national report will be released at the same time as the international report from the OECD. This report will be tailored to a general audience and will report the U.S. results relative to other countries. There will not be a report on the field test results.

B.5 Individuals Consulted on Study Design

Many people at OECD, Statistics Canada, and other organizations around the world have been involved in the design of TALIS. Overall direction for TALIS in the U.S. is provided by Mary Coleman, the TALIS Governing Board representative and National Project Manager at the U.S. Department of Education:

Mary Coleman

National Center for Education Statistics

U.S. Department of Education

[email protected]

References

OECD (2004), OECD Handbook for Internationally Comparative Education Statistics: Concepts, Standards, Definitions, and Classifications, OECD, Paris.

1 Following the advice of Statistics Canada, Westat used the number of estimated ISCED 2 students rather than teachers as the measure of school size. This is due to the uncertainty of the number of ISCED 2 teachers in certain types of schools (e.g., grade 9 teachers in grades 9-12 high schools) where the number of ISCED 2 students is a more accurate measure.